JavaScriptApplicationswithNode.js,
React,ReactNativeandMongoDB:
Design,code,test,deployandmanageinAmazonAWS
EricBush
“GivemesixhourstochopdownatreeandIwill
spendthefirstfoursharpeningtheaxe.”
-AbrahamLincoln
BLUESKYPRODUCTIONSINC.
Copyright©2018,EricBush
All rights reserved. No part of this book may be reproduced, stored in a
retrievalsystem,ortransmittedinanyformorbyanymeans,withouttheprior
writtenpermissionoftheauthor,exceptinthecaseofbriefquotationsembedded
in evaluation articles or reviews as accompanied by proper references to the
book.
Every effort has been made in the preparation of this book to ensure the
accuracy of the information presented. However, the information contained in
this book is sold without warranty, either expressed or implied. Neither the
book’sauthor,nor publisher,noritsdealersanddistributors willbeheldliable
for any damages caused or alleged, resulting directly by or indirectly by this
book.
Theviewsexpressedhereinarethesoleresponsibilityoftheauthoranddo
notnecessarilyrepresentthoseofanyotherpersonororganization.
Pleasesendanycommentstotheauthoratjsdevstack@outlook.com.
PaperbackISBN-10:0-9971966-6-1
PaperbackISBN-13:978-0-9971966-6-5
e-bookISBN-10:0-9971966-7-X
e-bookISBN-13:978-0-9971966-7-2
Contents
Preface
AbouttheAuthor
Introduction
PARTI:TheDataLayer(MongoDB)
Chapter1:Fundamentals
1.1DefinitionoftheDataLayer
1.2Datalayerdesignprocess
1.3IntroducingMongoDB
1.4TheMongoDBCollection
1.5TheMongoDBDocument
Chapter2:DataModeling
2.1ReferencingorEmbeddingData
2.2WhentouseReferencing
2.3ReferenceRelationshipPatterns
2.4AHybridApproach
2.5DifferentiatingDocumentTypes
2.6RunningOutofSpaceinaDatabase
2.7AccessControl
Chapter3:QueryingforDocuments
3.1QueryCriteria
3.2Projectioncriteria
3.3QueryingPolymorphicDocumentsinaSingleCollection
Chapter4:UpdatingDocuments
4.1Updateoperators
4.2ArrayUpdateOperators
4.3Transactions
Chapter5:ManagingAvailabilityandPerformance
5.1Indexing
5.2AvailabilitythroughReplication
5.3Sharding
Chapter6:NewsWatcherAppDevelopment
6.1CreatetheDatabaseandCollection
6.2DataModelDocumentDesign
6.3TryingOutSomeQueries
6.4IndexingPolicy
6.5MovingOn
Chapter7:DevOpsforMongoDB
7.1MonitoringthroughtheAtlasManagementPortal
7.2TheBlameGame
7.3BackupandRecovery
PARTII:TheServiceLayer(Node.js)
Chapter8:Fundamentals
8.1DefinitionoftheServiceLayer
8.2IntroducingNode.js
8.3BasicConceptsofProgrammingNode
8.4Node.jsModuleDesign
8.5UsefulNodeModules
Chapter9:Express
9.1TheExpressBasics
9.2ExpressRequestRouting
9.3ExpressMiddleware
9.4ExpressRequestObject
9.5ExpressResponseObject
9.6TemplateResponseSending
Chapter10:TheMongoDBModule
10.1BasicCRUDOperations
10.2AggregationFunctionality
10.3WhatAboutanODM/ORM?
10.4ConcurrencyProblems
Chapter11:AdvancedNodeConcepts
11.1HowtoScheduleCodetoRun
11.2HowtoBeRESTful
11.3HowtoSecureAccess
11.4HowtoMitigateAttacks
11.5UnderstandingNodeInternals
11.6HowtoScaleNode
Chapter12:NewsWatcherAppDevelopment
12.1InstalltheNecessaryTools
12.2CreateanExpressApplication
12.3DeployingtoAWSElasticBeanstalk
12.4BasicProjectStructure
12.5WhereitAllStarts(server.js)
12.6AMongoDBDocumenttoHoldNewsStories
12.7ACentralPlaceforConfiguration(.env)
12.8HTTP/RestWebServiceAPI
12.9SessionResourceRouting(routes/session.js)
12.10AuthorizationTokenModule(routes/authHelper.js)
12.11UserResourceRouting(routes/users.js)
12.12HomeNewsRouting(routes/homeNews.js)
12.13SharedNewsRouting(routes/sharedNews.js)
12.14ForkedNodeProcess(app_FORK.js)
12.15SecuringwithHTTPS
12.16Deployment
Chapter13:TestingtheNewsWatcherRestAPI
13.1DebuggingDuringTesting
13.2ToolstoMakeanHTTP/RestCall
13.3AFunctionalTestSuitwithMocha
13.4PerformanceandLoadTesting
13.5RunningLint
Chapter14:DevOpsServiceLayerTips
14.1ConsoleLogging
14.2CPUProfiling
14.3MemoryLeakDetection
14.4CI/CD
14.5MonitoringandAlerting
PARTIII:ThePresentationLayer(React/HTML)
Chapter15:Fundamentals
15.1DefinitionofthePresentationLayer
15.2IntroducingReact
15.3ReactwithonlyanHTMLfile
15.4InstallationandAppCreation
15.5TheBasicsofReactrenderingwithComponents
15.6CustomComponentsandProps
15.7ComponentsandState
15.8EventHandlers
15.9ComponentContainment
15.10HTMLForms
15.11LifecycleofaComponent
15.12TypecheckingyourProps
15.13GettingareferencetoaDOMelement
Chapter16:FurtherTopics
16.1UsingReactRouter
16.2UsingBootstrapwithReact
16.3MakingHTTP/RestRequests
16.4StatemanagementwithRedux
Chapter17:NewsWatcherAppDevelopmentwithReact
17.1WhereitAllStarts(src/index.js)
17.2Thehubofeverything(src/App.js)
17.3ReduxReducers(src/reducers/index.js)
17.4TheLoginPage(src/views/loginview.js)
17.5DisplayingtheNews(src/views/newsview.jsandsrc/views/homenewsview.js)
17.6SharedNewsPage(src/views/sharednewsview.js)
17.7ProfilePage(src/views/profileview.js)
17.8NotFoundPage(src/views/notfound.js)
Chapter18:UITestingofNewsWatcher
18.1UITestingwithSelenium
18.2UITestingwithEnzyme
18.3DebuggingUICodeIssues
Chapter19:Server-SideRendering
19.1NewsWatcherandSSR
Chapter20:NativeMobileApplicationdevelopmentwithReactNative
20.1ReactNativestarterapplication
20.2Components
20.3Stylingyourapplication
20.4Layoutwithflexbox
20.5Screennavigation
20.6Devicecapabilityaccess
20.7CodechangestoNewsWatcher
20.8ApplicationStoreDeployment
CONCLUSION
INDEX
Acknowledgements
Thisbookcoverssomephenomenaltechnologyframeworks.Oneofthoseis
Node.js. I am in awe of Ryan Dahl for his having conceived of it in the first
placeandforhishavingbuiltitontopofsomegreatworkbyothers.Thankyou
Ryan!Iwanttothankallthosewhohavecollaboratedonthemodulesthatmake
Nodesuchacapableplatform.Thisisopen-sourceatitsbest.
I would also like to thank TJ Holowaychuk for his contribution to node
through his work in creating the Express web framework. This really does a
greatjoborganizingandsimplifyingthecodeneededtoimplementaRestWeb
API.
IwanttothankthebrilliantpeoplebehindalloftheAmazonWebServices
infrastructure.Fromthosewhoenvisioneditinthefirstplace,tothosewhobuild
itandalsothosewhorunit24-hoursaday.Everyyearitaddsamazingplatform
services for all to take advantage of. This liberates application developers to
code up their applications to provide customer value and benefit from many
underlyingservicesthatdoalotoftheheavylifting.
AnothergreatpieceoftechnologyisthatofMongoDB.Iwanttothankthose
who develop and maintain MongoDB and also thank the Atlas team for their
makingiteasytohostMongoDBinaPaaSenvironment.
I acknowledge all the effort that is ongoing to build, maintain and evolve
ReactandReactNative.TheReactframeworkfromFacebookisaperfectfitin
theJavaScriptstack.
IexpressmylovetomysweetheartLoradel.Thankyouforfindingme.Iwill
cherishyouforever.
Preface
Manypeoplewillturntothefirstfewpagesofabooktodeterminewhether
or not it will meet their needs. If you are just now making a decision about
whetheryouwanttoinvestthetimeandmoneynecessarytobuildupyourskills
intheareaoffull-stackdevelopment,Iwillstatethatyouaredoingwelltohave
foundthisbook.Letmegiveyouabriefmarketingpitchaboutwhyyouneedto
pursueinvestinginfull-stackJavaScriptskills.
Technologytrendscomeandgo,butwhatthisbookcoversisreallyhereto
stay.TheJavaScripttechnologiesthatthisbookcovershavecometogetherina
unified stack, all using the JavaScript programming language. With this book,
you will not only learn about these technologies, but will also gain a longer
lastingfoundationalunderstandingofwhatathree-tierarchitecturaldesignis.
IfyouareevenremotelyinterestedinJavaScriptdevelopment,Iadviseyou
todedicatetimetolearnthesetechnologiesanditwillpayoff.Youwillhaveto
study hard, but it will be worth it. There is a huge development community
behindthisandyouwillbeingoodcompany.Manytopcompaniesarefinding
successdoingfull-stackJavaScriptdevelopmenttoday.
Levelofskillrequired
YouwillneedtohavesomebasicunderstandingofJavaScripttobeeffective
inlearningwhatthisbookcontains.Ifyoudon’talreadyknowthebasicconcepts
oftheJavaScriptlanguage,youcanuseonlinecontenttolearnthemoryoucan
buyabooktolearnthem.
Ihavewrittenthisbookforalllevelsofexperience.Ifyouareabeginnerand
needtolearnsomeofthebasicconceptsofathree-tierarchitecture,Ihaveyou
covered.Ialwaysleadoffwithsomematerialthatcoversthebasicconceptsthat
areinvolvedinthearchitecture.Idothissothatwhenyougettotheactualcode
implementation,youwillhavethisinsightandarenotjusthaphazardlywriting
code.
Thisbookpullstogetheralotofinformationthatyouwouldhaveahardtime
finding.Oneofmygoalswastogiveyouknowledgethatwillhelpyougetand
keepajob.Ihaveputmyselfinyourshoesandhavegivenyoutheinformation
thatyoureallyneedtogaintraction.
IfyouarealreadyfamiliarwiththebasicsoftheMERNstack,thisbookcan
getyoutothenextstageandactuallyleadyoutowardsprofessionalenterprise-
level development. I make sure to dive deep into each topic and show real
implementationanddeploymentdetails.Inotherwords,thisbookisnotacopy
oftheAPIdocumentationofeachframework.
For the experts in MERN development out there, this book may save you
timetryingtofigureoutwaystotestandsecureyourapplication.
Halfthe battleof learninganynew technologyislearning theadd-onsand
toolstogetitallworking.Thisiswherethisbookwillreallycomeinhandy.I
don’tjustdumpalotofcodeonyou,Itakethetimetoexplainthedevelopment
processandwhattoolsyoucanuse.
AWSandotherknowledge
This book utilizes AWS for its implementation. You get to learn about
configuring and deploying to that environment. As mentioned, this book also
contains substantial foundational instruction, such as details on security
measuresandinformationonhowtoperformtesting.
Choices
Thisiscertainlyagreattimetobeasoftwaredeveloper.Oneofthethings
thatmakesitsogreatisthemyriadofopen-sourceprojectsavailabletoleverage.
Thereareframeworksthatexisttohelpyouwitheverylayerofyourarchitecture
fromtheback-endservicestothefront-endGUI.Thereareliterallyhundredsof
options for technologies touse in your development of a full-stack JavaScript
application.
I will help you get started by narrowing things down to some of the
essentialsthatareneededtogetupandrunning.Iwillalsocovermoreadvanced
topicstohelpyoubuildqualityenterpriseapplications.
Ihadtomakesometoughdecisionsonwhichframeworksandtoolstoutilize
inthisbookandhavedonesobasedonthefollowingcriteria:
1. Itmustbesimpleandquicktogetvaluefrom.
2. Itmusthave wideadoption,with committedsupportfor
growthandfuturerelevance.
3. Itmusthavebroadplatformreach.
After mastering what is covered in this book, you can turn around and
evangelize full-stack JavaScript development as a reality today. It is great
alreadyandwillonlygetbetter.
Happycoding,
EricBush
NoteaboutapreviousbookIauthored:Basedontheaboveandmanyother
factors,ImadetheswitchfromAngulartoReactforthisbookandchangedthe
title.Manyotherchangesandadditionswerealsomade.
AbouttheAuthor
EricBushbeganhisprofessionalcareerprogramminginassemblylanguage
for embedded hardware microcontrollers for airplane flight systems. He then
worked developing CAD software on what were known as workstation
computers.WhenWindowscamealong,Ericstarteddevelopingsomeofthefirst
desktop applications for Windows. After that, he transitioned back to client-
serverapplicationsforWindowsServer.Finally,hemadethemovetoenterprise
cloud development in 2009 and is now focused on full-stack JavaScript
technologies,usingMongoDB,Node.jsandReact.
He has worked for some well-known companies including Walmart Labs,
Microsoft,Tektronix,MentorGraphics,Nike,Intel,andBoeing.
Ericworkedasafull-stackdeveloperfortheworldwideretailgiantWalmart.
HeworkedonbackendscalableJavaScriptNode.jscloudinfrastructureandalso
on front-end React code for the walmart.com website store details pages. The
backendworkservesuphigh-trafficproductsearchesandcartcapabilitiesused
by millions of people each day. The work included many of the HTTP Rest
endpoints in use. He was also involved in the DevOps work for Walmart
enterprise practices of scalability, quality, security, reliability, availability and
performance.
As a consulted at DocuSign in downtown Seattlehe was in the role of an
architect/developer and developed an internal enterprise application suite of
toolstoimprovetheefficiencyofemployees.Thisinvolvedfull-stackJavaScript
withHTML/AngularthattalkedtoabackendNode.jsserverthatintegratedwith
aMongoDBdatabase.
AsaconsultantthroughCrosslakeTechnologies(crosslaketech.com)heputs
his considerable breadth and depth of knowledge into helping companies
implement modern PaaS cloud architectures. Crosslake helps companies
envision, plan, architect, develop and execute engineering practices and
architectures to satisfy business and customer needs. Crosslake works to
transform and optimize software delivery. Crosslake also provides services for
technicalduediligenceformergers,acquisitionsandinvestments.
YoucanemailEricatjsdevstack@outlook.com.
Introduction
Thisbookpresentsthetechnologiesandbestpracticesthatspanacrossallof
thelayersofasoftwaredevelopmentarchitecture.Thecoreofthisbookfocuses
on specific JavaScript frameworks used to implement each architectural layer.
Everything is based on MongoDB, Express, React, React Native and Node. I
will walk you through the design and development of a sample application to
teachyouthebestpracticesinarchitecting,coding,testing,securing,deploying
andmanagingaRESTfulWebServiceandSPA(SinglePageApplication).
Thisintroductionsectionwilldefineseveralkeyterms,andintroduceyouto
thesampleapplication.Thisisimportantcontentthatyouwillneedbeforeyou
divedeeperintothematerialspresentedintherestofthebook.
Whatisadevelopmentstack?
Adevelopmentstackisthecollectionoflanguagesandtechnologiesusedto
constructthesoftwareapplication.Thesearethetechnologiesyouwouldpiece
togetherfrombottomtotop.Thisisdifferentfromwhatasoftwarearchitecture
is.Anarchitectureisapatternofcomponentsandlayersforbuildingsoftware.
This book utilizes a particularly popular set of technologies referred to as the
MERN (MongoDB, Express.JS, React and Node.js) development stack. The
architecturepatternfollowedisreferredtoasaSOAthree-tierarchitecture.
MongoDB, Express.JS, React and Node.js are called platforms, or
frameworks,becausetheytakecode,orsomeformofmarkup/configurationand
executeitatahigherlevelofabstractionabovetheoperatingsystemlevel.
TheMERNstackonlyspecifiesfourtechnologies.However,inreality,there
aremanymoretechnologiesthatcomeintoplay.Node.jsasaframeworkreally
gets its capabilities from modular plug-ins that extend its basic capabilities.
Platforms are usually extensible, meaning they are built to be extended in
functionalityby others inthe community thatprovide modules. Youwilllearn
about many of the important extensions that can be utilized to extend the
capabilitiesofNode.js,ExpressandReact.
Note: AWS does have database services such as DynamoDB, a NoSQL
databasetechnology.I havechosento develop withMongoDB instead. Thisis
becauseofitsrichsetofcapabilitiesanditspopularity.MongoDBisstillhosted
inAWSasaPaaSservice,eventhoughitisnotprovideddirectlybyAWS.
Thethree-tierarchitecture
No matter what you are constructing, it is a good idea to modularize and
buildwithlayerssothatyoucanmoreeasilyassembleeverything.Ifyouwere
buildingahouse,youwouldnotbuilditasonejumbled-upmess.Instead,you
wouldfirstlayafoundation,andeventually,constructtheatticandroof.Eachof
the needed modules would fit together: flooring, walls, heating, plumbing,
electrical,etc.Softwareconstructioncanbethoughtofinasimilarway.
Architecting software in layers along with writing code as modules (also
referred to as components) helps with testing, debugging, and incorporating
futureenhancements.Thelatestbuzzwordintheareaofsoftwareconstructionis
“microservices.” You may also have heard of the older concept of a service-
oriented architecture (SOA). These are patterns that encourage layers of
separation.
Oneprovenapproachtodevelopingasoftwareapplicationistodivideitup
intothreedistinctlayers.Thethreelayers(ortiers)arenameddata,service,and
presentation.Youcanalsoutilizepatternssuchasmodelviewcontroller(MVC)
withinathree-tierarchitecture.
Definitionsofthethreetiersarelistedbelow.Anassessmentisprovidedfor
whereeachMERNtechnologyfitsintothoselayers.Whilethislistgivesyouthe
briefestofintroductionstoeachofthese,thethreepartsofthisbookgointothe
detailsofeachlayer.
DataLayer(MongoDB):
This layer is where you persist your data in a DBMS (Database
Management System). Data can be stored, retrieved, updated, or deleted.
MongoDBiscategorizedas a“document-based”database.Thisbook will
discuss a JavaScript API used to interact with MongoDB. Amanagement
portalcalledtheAtlasportalandatoolnamedCompasswillbeused.
ServiceLayer(Node.js+Express.js):
This layer is where a web service API is exposed to contain your
businesslogicandisthedoorwaytothedatastoredinthedatalayer.The
service layer performs workflows that require more complicated
computations and sequencing of operations. These might be HTTP
endpoints or code that is scheduled to run periodically. Node.js was
specifically designed for writing scalable server-side web applications.
Express.jsis used as a Node add-onmodule to simplify the buildingof a
web service layer. Express has support for HTTP request routing and
sendingbackofresponses.
PresentationLayer(React/HTMLandReactNative):
Thislayerprovidesthecapabilitytointeractwithanddisplaydata.This
could be on a computer or mobile device of some type. React provides a
waytocodeupcomponentsthatrenderaUIbyformulatingtheUIcontrols
thatdrivethedisplay.
Note:Inmyopinion,themiddleservicelayerchoiceofNode.jsissomething
foundationaltoaJavaScript-baseddevelopmentstackandcouldnotbeswapped
out. The other choices of MongoDB and React could be swapped out and
replaced with some other similar technology choices. For example, since
MongoDB is accessed through a Node module, you could easily find another
solutionusingwhateverdatabaseyouprefer.Reactisthehighestlayerandcould
bereplacedwithanynumberofframeworksthatallowyoutomakeHTTPweb
requeststotheNodeservicelayeranddothedatabindingwitheitherclientor
server-siderendering.
UsingJavaScripteverywhere
Youunderstandfromthe booktitle,that100%ofthe developmentdonein
thisbookisinJavaScript.Thereareconsiderablebenefitswithhavingonesingle
language used in all three architecture layers. For one thing, the code will all
lookthesameandbesimilarlyrefactoredandmaintained.Youcanalsobenefit
fromusingthesametestingframeworkacrosslayers.Someofthelibrariesyou
downloadcanevenbeusedinbothReactandNode.jscode.
JavaScript has huge momentum with lots of online resources and tons of
available open-source code. You will soon become good at searching online
before you start coding anything. Always search first, just in case there is a
moduleouttherethatwilldowhatyouneed.
Note:YoucandecidetouseTypeScriptwithyourfull-stackcoding.Thecode
fortheusageofNode.jscanstillbethesameasfoundinthisbook,aswellasthe
usageofReactandMongoDB.Theremaybeminordifferences,butthosecanbe
researchedontheinternet.
Note: JavaScript Object Notation (JSON) will be used as the data-
interchangeformat.ThisisanexcellentfitwithaJavaScriptapplication.
Usingpubliccloudinfrastructure
Every piece of code and all data storage should be hosted on public cloud
infrastructure.Thisgivesyoucharacteristicslikefastdeployment,lowcost,and
elastic scalability. This book shows you how to construct an application that
hostseverythinginAWS.
Asfarascloudhostingstrategies,theycanbroadlybebroadlysplitupinto
three categories. These are IaaS, PaaS, and SaaS. The “aaS” part of each
acronymstandsfor“asaService”.
InIaaS,the“I”isfor“Infrastructure”andmeansyoucanutilizethelowest
levelvirtualmachines (VMsor containers)and havecompletecontrol ofwhat
operatingsystemsandsoftwareyoudeploy.
InPaaS,the“P”isfor“Platform”.Thisvariationmakesyourlifeabiteasier
as a developer. It still gives you the flexibility you need over machine
deployment and scaling, but it frees you from the day-to-day maintenance of
machines. You could go through the added work of using IaaS to install and
manage your own Node.js service, or use a PaaS service instead that is much
simpler to setup and operate. After all, you want to spend your time on
applicationdevelopment,notoninfrastructureprovisioningandmaintenance.
InSaaS,the“S”isfor“Software”andindicatesacompleteofferingsuchas
Salesforce.com,AzureSharePoint,orAmazonWorkMail.Thinkofthisasbeing
applications that formerly would have been individually installed on your
machineatworkorhome,butthatnowcanberunningonamachineinthecloud
andaccessedfromanywhere.
Note:Sincecustomersusingacloudservicedon’tknowwhenchangeswill
happen to the service, you want to make sure they have uninterrupted access.
For example, with Elastic Beanstalk as your hosting PaaS, your deployment
updatescanbe“rolling”sothatyourserviceisnevertakendown.Youwilllearn
more about this. AWS also handles OS and hardware upgrades for you in a
rollingmannerthatkeepsyourserviceupwhilethathappens.
MicrosoftVisualStudioCode
Ifyouareoutintherealworldtryingtomakealivingbywritingsoftware,
therealityisthatyouhavetobeabletocombinemanytechnologiesatonceand
targetmanydifferentplatforms.Centraltothisisthecodeeditorthatyouchoose
touse.VisualStudioCode(VSCode)isagreattoolforeditingyourcode.This
isthe tool I used to code and debug the sample applicationused in this book.
Youcancertainlychoosewhatevercodeeditoryoulikeandarenotrequiredto
useVSCode.
VSCodeoffersaricheditingenvironmentaswellasintegratedfeaturesfor
sourcecodecontrol anddebugging. VSCode alsohasthe capabilityto launch
tasksusingtoolslikeGulp,withoutneedingtojumptoacommandlineprompt.
These tools can be used to automate the build and testing steps that you run
frequently.
UsingVSCode,youcancreateaproject,runitlocallyonyourmachine,and
have access to IntelliSense, debugging, git commands and app publishing. VS
CodecanbeinstalledonMacOSX,andLinux,aswellasonWindows.
If you are starting from scratch and don’t have any development tools
installed,goaheadandinstallthefollowing:
VisualStudioCode.DidImentionhowcoolthisis?
Git.Thisisusedforsourcecodecontrol,butitcanalsobethemeansof
deployingtoAWSinfrastructurethroughGitHubandothertools.
Node.js from https://nodejs.org/. Get a stable version that is labeled
LTS. This will also install NPM (a package manager for JavaScript
projects)atthesametime.
TheNewsWatchersampleapplication
ThecodingconceptsinthisbookareillustratedusingasampleapplicationI
called NewsWatcher. The sample application will help you understand how
everythingcomestogetheracrossallthreelayersoftheapplicationdevelopment
stack. I will now walk you through the capabilities and architecture of the
sampleapplicationcalledNewsWatcher.
The basic capability of NewsWatcher allows a user to set up keywords to
filterthroughnewsstories.Theusercanview,shareandcommentonthenews
stories.Itwillinitiallybedevelopedasawebsitetorunonadesktopormobile
device browser and then the UI will also be written to work as a native
applicationforiOSandAndroid.
It is important to spend a short amount of time on the vision for what
NewsWatcher was meant to be. I needed a vision statement and also some
sketchesofwhatitshouldlooklike.Aroughplanforthedevelopmentwaslaid
out from there, including a prioritized backlog of features. I happen to prefer
followingtheKanbanprocesstoworkthroughtheworkinabacklog.Hereisthe
visionstatementthatIputtogetherforNewsWatcher:
VisionandValuepropositionStatement
Users of NewsWatcher will get news personally served up to them.
NewsWatcherwillbetheirtrustedadvisortoalertthemandsaveimportant
information,withouttheirneedingtoconstantlyscanendlessfeedsofnews
stories.Userscandothingslikeadjustfilterstogetjustthenewstheyare
interested in, see what their friends are looking at, and share comments
aboutnewsstories.
Thisvisionstatementisthelonger-termgoalofwhattoshootforandwould
takemanyiterationstorealize.Fromthisvisionstatement,Iwasabletodistill
thevisiondowninto alistof featuresthatcan beiteratedover.HereiswhatI
cameupwithforNewsWatcher.ThelistcontainssomerequirementsandImay
havesnuckinabitofimplementationdetailalso.
Prioritizedfeaturelistthatfulfillsthevision:
1. Icansetupmultiplenewsfilterstoholdnewsstories.
2. Icangivemyfilteratitletoidentifyit.
3. I can set up keywords for each filter with Boolean operations of
ANDandOR.Forexample,“TreatmentsANDinsomnia”,“DogsOR
Cats”.
4. ThefiltersareperiodicallyscannedbytheNewsWatcherbackend
service to search for news stories that match from a central pool of
collectedstoriesfromanewsserviceprovider.Client-sideprocessing
will be freed up. Finding of stories proceeds even while devices are
turnedofforunabletoconnecttotheinternet.Theserver-siderunsa
periodicpullofstoriesfromanewsfeedservice.
5. Newsstoriesinfolderscanbescrolledthroughandclickedonto
takemetothestorycontent.The20mostrecentstoriesareshownfor
eachfilter.
6. UponopeningNewsWatcher,Iseethehomepageofnewsstories.
7. Icanclickonandopenanewsfoldertoseethestoriesinit.
8. Storiesinfilterfolderscanbesavedtoanarchivefolder.
9. IcandeleteindividualsavedstoriesIselect.
10. Icandeletethefilteritselfwithallofitsstories.
11. My account settings are stored server-side, but cached on the
deviceclient-sidealso.
12. IcandeletemyNewsWatcheraccount.
13. TherearegloballysharednewsstoriesinwhichIcanshareastory
tobeseenbyallNewsWatcherusers.AllotherNewsWatcheruserscan
see and comment on these stories. Stories will be kept for a week
before being deleted. A maximum of thirty shared stories can exist;
newoneswillbumpolderonesoffbeforetheweekisup.Anygiven
user is limited to sharing five stories a week. There can be thirty
commentsperstorykept.Offensivelanguageincommentsisblanked
out.
14. IcanviewmynewsonwhateverdeviceIamloggedinwith.
15. Theloginexpiresandrequiresmetologinagainperiodically.
16. Two-factor authentication requires a text message code to be
entered.
17. I can set actions on news folders to alert or email me when new
entriesarefound.
18. Icansetlimitsonwhennewsalertsaresenttome.Forexample,
waitatleasttenminutesbetweenalerts.
19. IcansettimesduringthedaywhenIdon’twantnewsalertssentto
me.
20. I can set NewsWatcher to download stories to my device when
connectedtoWi-Fi.
21. Icanuseasearchentryboxtodoasearchofallloadedstories.
22. IcandesignatetheorderIwanttoseethefiltersin.
23. I can email stories to others or share them via Twitter and
Facebook.
24. iOS and Android phones have a Native app available in the app
stores.
This book will only implement the core functionality of NewsWatcher in
ordertoproduceaMinimalViableProduct.Noteverythingwillmakeitintothe
initialiteration,butitisnicetohaveabacklogreadytodrawfrom.
Wireframeprototype
The following wireframe images were created in PowerPoint. PowerPoint
hasagreatsetofstoryboardtemplatesyoucanusetopiecetogetheraUIimage
and make realistic looking prototypes. There are also many nice websites you
canusetosketchoutaUIwith.
TocreateaPowerPointprototypeimage:
1. OpenPowerPointandcreateablankslide.
2. Click the Storyboarding tab, then click Storyboard Shapes on
theribbontoopentheStoryboardShapeslibrary.
3. Start creating your prototypes by dragging shapes from the
StoryboardShapeslibrarytoyourslide.
Figure1-NewsWatcherwireframes
Note:IfyouareusingPowerPoint2013,youneedto haveinstalledVisual
Studio 2013 or later, or the Team Foundation Server Standalone Office
Integration 2015 on the same machine to create and modify storyboards.
PowerPoint2016isinstalledwitheverythingyouneedforthisfeaturetowork.
Apeekattheresult
HereareafewscreenshotsofwhattheactualWebUIendeduplookinglike
onamobiledevice.Itdoesn’tlookquitelikethewireframes,asanygooddesign
willevolve.
Figure2-NewsWatcheractualresultingUIonamobilephone
You can try the completed sample browser app on your mobile device or
desktopby going tohttps://www.newswatcher2rweb.com. Download the React
NativeappfromGooglePlay.
AllofthecodeforthesampleapplicationcanbefoundonGitHub.Youcan
download a ZIP file from https://github.com/eljamaki01/NewsWatcher2RWeb.
The React Native code is found at
https://github.com/eljamaki01/newswatcher2RN.
Apeekatthearchitectureanddeploymenttopology
The three-tier architecture of NewsWatcher is depicted in the following
diagram.Thearrowsshowwhatpartsofthearchitecturecallwhatotherparts.
Figure3-NewsWatcherarchitecturediagram
The machine topology required to implement a three-tier application can
vary widely and may even change over time. This is because the topology is
reallyafunctionofthescalingthatyourapplicationneedstoachieve.Youwould
certainly not start off with a topology that could scale to millions of users. It
wouldbeunnecessarilycomplicatedtoprovisionandmaintain.Youdoonlypay
forwhat youneed, however,some things wouldbe inplace thatwill stillcost
youmoneythatyouwouldotherwisenotneed.
Thefollowingdiagramgivesyoualookatastartingpointforanarchitecture
topology. This topology would give you a fair amount of scaling to handle a
largenumberofusers.
Figure4-NewsWatcherservicehostingtopology
PARTI:TheDataLayer(MongoDB)
Thefirstpartofthisbookwillnowinstructyouonwhatadatalayerisand
howtoimplementoneusingMongoDB.ThiswillbehostedusingDBaaS/PaaS
inAWSusingtheMongoDBInc.Atlasoffering.
Theveryfirststepincreatingadatalayerinvolvesthemodelingofthetypes
ofdatayouwillwanttostore.Afterthat,youwilllearnwhatMongoDBisand
howitcanbeusedtoimplementadatamodel.
To finish out the first part of the book, you will actually construct a data
layer that will support the needs of the NewsWatcher sample application. You
will end up with a fully capable data layer that can be utilized by the service
layeroftheapplicationarchitecturebeingbuiltinthesecondpartofthisbook.
Inordertogivefullcoveragetothetopic,Iwillalsocoverwhatitmeansto
managetheday-to-dayoperationsofaMongoDBdeployment.
Chapter1:Fundamentals
Thischapterpresentstheconceptsofabackenddatastoragesystem.Iwill
show you what one is composed of and what capabilities are essential. After
lookingatbackendstoragesystemsingeneral,Iwilldelveintothespecificsof
MongoDBandshowhowitisintegratedintotheNewsWatcherarchitectureto
fulfillthebackenddatastoragerequirements.
Note: This book is about full-stack development. While you will be shown
how to get MongoDB set up in a PaaS environment and learn some basic
development and monitoring, topics like what it takes to fully administer the
serviceare out of scope forthis book. Please seethe MongoDB and theAtlas
helpresourcesformoreinformationonadministrativetopics.
1.1DefinitionoftheDataLayer
The data layer of a software application architecture provides for the
persistent storage of information. Anything of importance can be stored there,
such as customer account information, inventory, orders, audit logs, tax
computation tables, and anything else that can conceivably be stored in
electronicform.
The term database management system (DBMS) is used to describe the
commercial offerings that implement a data layer at the lowest level. Each
availableDBMShascapabilitiesthatdifferentiateitfromtheothers.MongoDB
fromMongoDB,Inc.isonesuch DBMSsystemthatisdevelopedas anopen-
source project as well as being a commercial offering. See
https://www.mongodb.com/ for their various offerings and capabilities.
MongoDBAtlas(https://www.mongodb.com/cloud/atlas)isacommercialPaaS
offeringfromMongoDBInc.Youwillalsofindothercompanies,suchasmLab
(https://mlab.com/)thatalsohostversionsofMongoDBinaPaaSenvironment.
DBMScapabilities
DBMSimplementationscan be placedinto severalgeneral categories.One
possiblewaytocategorizethemisasfollows:relational,key-value,hierarchical,
object-oriented,ordocument-based.SomeDBMSscanalsobesaidtobeofthe
“NoSQL”type.Eachcategoryexistsforaspecificreasonandyouwouldwantto
look at your specific needs and choose the particular DBMS technology that
fulfillsyourneedsthebest.
A DBMS will provide the physical storage medium where the data resides
and should even withstand a power failure. In a cloud-hosted DBMS, data
eventuallymakesitswaytobeingstoredonnon-volatilecloudstoragedrivesin
asecuredatacenter.Datastoragemightevenbegeo-replicatedbetweendistant
datacentersforredundancyandload-balancing.
A DBMS will provide for the storage and access features for the creation,
retrieval, updating, and deletion of data. The acronym “CRUD” is commonly
usedtorefertotheseoperations.ADBMSwilltypicallyprovidethefollowing
features:
Datastorage
Transactions
Attribution
Auditing
Authorization
Dataaccess
Indexing
Encryption
Notification
Programmability
Schematization
Security
Transformation
Validation
Eventhough theDBMS itselfprovidessometype ofprogrammatic access,
thereisoftenanadditionallayerontopthatisreferredtoasaDataAccessLayer
(DAL). There are community written DALs as well as those that are
commerciallyavailable,oryoucanevenwriteyourown.TheDALwillcreate
an abstraction layer that hides the complexities of the data storage technology
and may even allow you to switch to a different backend DBMS without
affectingtheupperlayersofyourapplication.
ADALcangreatlysimplifyyouraccessbycreatinganobjectstructurethat
mightnotevenexistintheactualdatastoragesystemitself.Forexample,some
DBMSs don’tprovide schematization of data. If you need that, you can get it
through a DAL. The following image shows all the sub-layers within the data
layer:
Figure5-DataLayerwithsub-layers
1.2Datalayerdesignprocess
Before storing any data in your data layer, you need to create models for
whatthatdatashouldlooklike.Knowingwhatyouwouldliketostoreisthefirst
step to take. With that information determined, you can model what the
structural form of the data will be. This will require some thoughtful design
work to be able to organize your data in an efficient way. You will need to
diagramoutamodelforyourdatatoaidinunderstandingallthenuancesofthe
recordtypeswiththeirpropertiesandrelationships.
User interfaces come and go, but backend data systems seem to live on.
Often the data in a data layer outlives the applications that were written to
expose it. It can sometimes be hard to go back and modify your data storage
afteryouhavestartedusingit,sotakeyourtimeanddesignitforfuturegrowth.
Note: Even when the lifetime of the DBMS technology platform is finally
reached,thedatayouhavemayremainvaluable.Ifyoudecidetomigratetoa
newer DBMS technology, the data can be exported from your old DBMS and
thenbere-importedintoyournewDBMS.Mostsystemsprovidecapabilitiesto
exportandimportdatainbulk.Somesystemsevenprovideanongoingsyncing
capabilityforthetransferandtransformationofthedata.
Datalayerplanning
Makesuretoutilizeexpertsalongthewaybeforerollinganythingoutintoa
production environment. The following questions are crucial as you work
throughtheinitial datalayerdesign. Theanswers tothese questionsshouldbe
carefullydetermined.
Datalayerplanningquestionnaire:
Is the data shared? If so, how is it shared between customers,
applications,andprocesses?
Ismulti-tenantstorageok?Whatisolationofdataisnecessary?
Whatareyourdatasecurityandprivacyrequirements?
Willyoubestoringdatathatisclassifiedashighbusinessimpact
(HBI) or storing any personally identifiable information (PII)? If the
dataiscompromised,whatarethelegalramifications?
Doyouneeddataaccessrolestocontrolaccesstothedata?
Arethereanyperiodicprocessingjobsthatrunthatwillaccessthe
data?
Willparallelaccesstoasinglerecordcauseconcurrencyissues?Is
theresometypeoflockingorserializationofaccessrequired?Would
requestqueuingoroptimisticconcurrencycontrol(OCC)suffice?
Doyourequiretransactionalcapabilities?
Can writes be asynchronous or do you need immediate
synchronousacknowledgmentonawrite?
Isalagtimebetweenwriteandavailabilityofthatdataforaread
operationacceptable?
What is the service level agreement (SLA) requirement for each
CRUDoperation?
What are the access volume and rates per minute? Will there be
burstsofactivityoristheactivityevenlydistributedacrosseachday?
Whatisthesizeofthedata?Howmanyrecordsandhowlargewill
theybecome?
How many users will access the data? What will be the needed
datacapacitypercustomer?
Doyouanticipatethatthestructureofyourdatawillchange?
Doyouneedtokeeparecordofeachdataaccess,suchaskeeping
anaudittrailofaccessesandchanges?
Willyouberunningdataminingandbusinessintelligenceanalysis
overthedata?
Isthereastrictdataschemathatneedstobevalidatedagainst?
Whataretherecordtypes,contentsandrelationships?
What system dependencies are involved? How is the data
transferredandhowisitcombinedandverified?
Areyoustoringmediadatathatistypicallyinbinaryfileformsuch
asphotos,moviesetc.?
1.3IntroducingMongoDB
MongoDB is classified as a NoSQL database. This means that it is non-
relational and non-schematized. There is no central schema catalog or data
recordstructuredefinitionrequired.Thatcanexist,however,itisnotrequired.
Database records can basically be free-form. Don’t get too worried about the
unstructured nature of MongoDB records, you will learn how to set up a data
modelandhowtovalidatedatabeforeitisstoredinyourdatalayer.
MongoDBisalsoclassifiedasbeingdocument-based.Itisadocument-based
database because it conceptually stores JavaScript Object Notation (JSON)
documents. I say conceptually, because it does not actually store JSON
documents directly, but instead stores an internal binary representation.
DocumentsarestoredinBinaryJSON(BSON)form.
Document-basedstoragewithMongoDBreallyhitsasweetspotformodern
applicationneeds.Itgivesyouthebestofscalingandperformance.Becauseit
canbepurchasedasaPaaS-hostedservicethatrunsonAWSinfrastructure,itis
aneasytomanageenvironment.Youcan,ofcourse,installandrunityourself.
Note: If you look at the database offerings of AWS, you will find several
choices for data storage in the cloud, including DynamoDB, RDS/Aurora,
RedshiftandS3.Eachofferingexists tofulfilldifferentrequirements, andeach
hasitsownadvantages.MongoDBisofferedthroughMongoDBInc.
CompanyssuchasmLaborMongoDBInc.offerPaaSsolutions.TheAtlas
product from MongoDB Inc. can be installed as a service and run in cloud-
hostedservicessuch asAWS.Throughthe Atlasmanagement portal,you sign
up to use MongoDB and then your database cluster (replica set or sharded
cluster)isdeployedtocloudinfrastructureonyourbehalf.
Note: MongoDB itself is built on top of another open source component
called WiredTiger that is a data storage engine. This is something that is not
necessarytoknowabout.MongoDBisallthatyouwillinteractwithdirectly.
BenefitsofMongoDBwithAtlasPaaS
Figure 5 in the prior section showed an illustration of the data layer. This
includedthreedifferentsub-layers.Thegreatnewsisthatyougetallthreewith
MongoDB and its API access. Here are some compelling features to consider
whenevaluatingthebenefitsofMongoDB,especiallythroughtheAtlasoffering
thatusescloudinfrastructure:
Elastic storage capacity: You simply dial up and down your
storagecapacitybymovingtoahigherperformanceplanorbyadding
morecapacity.Youcan deleteinfrastructurethatis nolonger needed
and then not be charged for it anymore. You can configure your
resources through code or through a management portal. There is
practicallyunlimitedgrowthfor“pay-as-you-grow”storagecapacity.
Elasticperformance: You simply move your performance up or
downbyselectingadifferentplan.Toachievehigherthroughput,you
canpayforthehighestperformancetierandgetashardedclusterand
thehighestIOPS.
PaaS:MongoDBonAWSthroughAtlasisaPaaSoffering.Allthe
machinemanagementofsoftwareandhardwareupgradesarehandled
foryou.
APM:Capabilitiesformonitoring,alerting,scalemanagementand
backupskeepyouincontrol.
Features: Besides creating, reading, updating and deleting
documents, there are many features to use under the right
circumstance. Features such as views, indexing, replication, change
streams, sharding, aggregation, capped collections, TTL indexes,
accesscontrol,encryption,ACIDtransactionsandmuchmore.
Auto-Replication: Data is automatically stored in redundant
copies of the database. The replications are there for your safety, to
ensure availability through failover. You can also set up automatic
backupstohappen.
Security: MongoDB has an audit log to track all database
operations. For added security, you can have an SSL connection to
your database. MongoDB also has an encrypted storage engine
availabletoprotectdata.
TheAtlasMongoDBofferingonAWSisreadytomeetallyourbusinessand
customerneeds.Ithasagreatfeaturesetandisincreasingincapabilitiesallthe
time.
Tryitout
Ifyoureallywantedto,youcoulddownloadMongoDBandrunitonyour
localmachine.Youcouldalsodotheworktosetupacloud-hostedVMorusea
DockercontainerwithMongoDB.MypreferenceistouseMongoDBasacloud-
hostedPaaSsolution.
MongoDBInc.makesiteasytosignupforaplanthatthenhostsMongoDB
onyour choiceofAmazon AWS, MicrosoftAzure, orGoogle CloudServices.
ThisistheapproachIhaveusedforthisbook.Youwouldhavetodoyourown
researchifyouwanttodepartfromthat.Therestofthisbookiswrittenwiththis
approachinmind.
It takes just a few clicks to and be up and running with MongoDB as a
hosted service in AWS. You don’t need to worry about the daily details of
managingthesoftwareupdatesandmachinehardwaremaintenancetokeepitup
andrunning.Thereisnoneedtowasteanytimedealingwithhardwarefailures,
suchasthereplacementofdrivesornetworkcards.
Youenjoythebenefitofhavingmachinereplication,scaling,load-balancing,
failover, and backup. With the scaling of MongoDB, you only pay for the
storageand performance you desire. Youpay for what you use and can easily
scaledownwhenyounolongerneedasmuchdatastorageorperformance.
You are free to concentrate on the aspects of your application that deliver
value to your customer and increase ROI for your company. There are many
benefitswithPaaScloudinfrastructure.
Payforperformance
As previously mentioned, you only pay for what you need. As with any
cloud infrastructure, if you have an immediate need, you can pay for higher
performance machines to run MongoDB. Thankfully, there is a free option
throughAtlasforyourinitialinvestigations.
Youcanalsopayforyourhigherscalingcapabilitybypurchasingaplanthat
comeswithahigherstorageallotmentandisconfiguredwithmoremachinesfor
high-availabilityandauto-failover.Ifyouneedtoscaleuptomorestorage,you
cankeepaddingdatabaseclustersasneededandspreadyourdataacrossmore
machineswithsharding.
Youalsohavetheflexibilitytopayformorethanoneplanatthesametime
andputdataonthemoreexpensiveplanthatneedsthehighestthroughput,while
otherdatacanbeonthelessexpensiveplanofstorage.Youcanchangethings
liketheinstancesize,replicationfactorandshardingscaleasneededatanytime.
Insomecases,aconfigurationchangewillstillrequiredowntime,suchasa
change in the instance size. This downtime would happen while the primary
serverisbeingmigrated.Butthistimeshouldbeunderaminute.SeetheAtlas
documentationformoredetails.
MongoDBstructure
WithyourAtlasaccount,youcancreateoneormoredatabaseclusters.These
clusters then have databases that serve as the containers for what are called
collections.Databasesholdcollectionsandtheyalsoholdyourindexesanduser
accounts.Collections,however,arewhatcontainyourdataintheformofBSON
documents.
Each database can have a set of users with specific permissions. You can
controltheusersofyourdatabaseandgiveindividualsreadaccessandalsowrite
access.ThisdoesnotmeanthateveryonethatconnectstoMongoDBthroughthe
NewsWatcherapp needs theirown useraccount. Generalprogrammatic access
happensthroughyourservicelayer.Specificdataaccessiscontrolledthrougha
middle-tier login mechanism. That information will be covered later in this
book.
Note:Youcancreateuseraccountsattheclusterlevelthatworkacrossall
databasescreatedonthatcluster.Anaccountiscreatedwhenyoufirstcreatethe
cluster,foradminprivileges.
Thefollowingillustrationisanoverall visualrepresentationthatshowsthe
differentresourcesthatarepartoftheMongoDBmanagedplatformserviceand
howtheyrelatetoeachother.Theimportantthingtounderstandisthatyoucan
havemultipledatabases,witheachdatabasehavingmultiplecollectionsinit.
Figure6-MongoDBresources
1.4TheMongoDBCollection
A collection is a container for storing documents and is the main resource
withwhichinteractionshappenprogrammatically.YoucanwriteJavaScriptcode
inyourNode.jsservicelayerforoperatingonthedatainthecollection.
Collections are a convenient way to separate out data in a MongoDB
database. They collectively share whatever the limit of total storage is for the
planyouarepayingfor.
Towritedataintoacollection,youwillwanttouseoneoftheAPIsavailable
todothat.YoucanalsowritedatausingtheMongoDBcommandlinetool,or
theCompassUIapplicationthatcanbeinstalledonyourmachine.Youcanadd
any documents by hand. Most likely, you will want to use one of the code
languageAPIsavailable.
1.5TheMongoDBDocument
Everydatabaseeverinventedisfundamentallydesignedtostoredataasaset
of records. Each individual record contains information that is useful for later
retrieval.Forexample,inarelationaldatabase,therecordisintheformofarow
in a table. For example, you might have a table of customers where each row
represents a single customer with their name, age, and email as part of their
record.InAWSS3,arecordtakestheformofdatawithitsdescriptivemetadata.
InMongoDB,asinglerecordisrepresentedbyadocumentinacollection.
The following image illustrates how you can have multiple collections, with
each collection containing multiple documents. Each rectangle is a document
below:
Figure7-ExampleMongoDBcollectionswithdocuments
Documents in this example, are represented as JSON with top-level curly
braces enclosing each document. This is how they are shown textually, even
thoughtheyreallydonotappearthatwayinthedatabaseitself.
Thissimpleconceptofstoringdocumentsinadocument-baseddatabasehas
many advantages. JSON is easy to formulate and consume in code. Another
benefitisthatitcancontaincomplexhierarchicaldatainembeddedstructures.It
alsosupportsarrays.
Datatypesinadocument
TheprimitivedatatypessupportedinaMongoDBJSONdocumentarethe
sameonesthatareavailableinJSON.Theseare:
Array
Boolean
Null
Number
Object
String
Inaddition,thereisanextendedsyntaxinJSONthatcantaketheaugmented
BSONdatatypesandpreservethem.SomeoftheaddeddatatypesofBSONare
Date, Binary data, 64-bit integer and many others. You will eventually be
accessing MongoDB through code and will be using JavaScript objects with
theirsupporteddatatypes.
Iwillnowgiveyouafewtipsforstoringmediadataandcurrencyvalues,as
therearesomeinterestingtechniquestoutilize.
Oneoftheseissuesiswiththestorageofbinarydata,especiallymediadata
such as photos and movies. Media data cannot realistically be inserted into a
JSONfile.YoucouldusetheBSONbinarydatatype,butthemainproblemyou
would encounter would be that the maximum BSON document size is 16
megabytes.Amoviewouldsimplynotfitintoonesingledocument.Itisbestto
storebinarydatainaseparatestoragesystem,suchasAWSS3,andreferenceit
fromaMongoDBdocument.
Another issue is the representation of currency values, such as US dollars
and cents. The problem comes when you attempt to perform floating point
precision storage and arithmetic operations, such as might be required for
investment calculations. If Node.js were to calculate and print 0.1 + 0.02, the
result would be 0.12000000000000001. You are not even safe in using the
MongoDB 64-bit floating point datatype. This is due to the rounding errors
inherentinCPUsbecauseofhowtheyrepresentfloatingpointnumbers.
One solution is to multiply all your monetary values by a constant scaling
factor. Forexample, if you weregoing to store the valueof $1.99, don’tstore
1.99,butinsteadmultiplythatby1000andstoreanintegervalueof1990.Then
youcanperformmathonthoseintegervaluesandconvertbackfordisplaywhen
needed.
TheJSONdocument
Onedistinguishingcharacteristicofadocument-baseddatabaseisthatitcan
storecomplexhierarchicalobjectswithoutanypredefinedschema.This means
thateverydocumentinacollectioncouldhaveadifferentstructure.Thiscould
beconsideredbothanadvantageandadisadvantage.
Therearemanyadvantages tohaving aschema-less database.The obvious
oneisthatyoudon’tneedtospecifytheJSONformatinadvance.Thismeans
that there is no need to specify datatypes or have restrictions on them. It is
simplyamatterofcreatingtheJSONwiththename/valuepairsthatyoudesire
andtheninsertingthatintoacollection.HereisasimpleJSONdocument:
{
"_id":ObjectID("59612a3dc17c5416d0a33041"),
"myNumber":99,
"myString":"Hithere",
"myBool":true
}
Note:Irefertothename/valuepairsintheJSONDocumentas“properties”,
sinceitssyntaxis soclose tothesyntaxof theproperty inaJavaScript object
literal.TheMongoDBonlinedocumentation,however,usesthetermfield/value
forthepairs.
IfyouopenoftheMongoDBInc.Compassappandlookatsomedocuments,
youcansee eachinthe UIwithall theirpropertiestoexplore. Inthischapter,
documentswillbepresentedinthe textualJSONformsinceyoucan visualize
thembetterthatway,andifneeded,importthemintoacollectioninthatform.
Once you get to the chapter on Node.js and are using the API to
programmaticallyinteractwithMongoDB,youwillseeJavaScriptcodewiththe
objectliteralsyntaxused.Justbeawareofthatswitch.Asanexample,theobject
literal syntax equivalent of the prior JSON example would be as follows
(MongoDBwillprovidetheidpropertyuponinsertion):
varmydoc={
myNumber:99,
myString:"Hithere",
myBool:true
};
Note: There is one restriction on JSON documents to make them work in
MongoDB.Therestrictionisthatthepropertynamesinasingledocumentmust
notbeduplicated.Thismakessensebecause,ifyouweretoqueryandaskfora
given property and it existed twice, that would be a little confusing. The core
MongoDBstoragesystemthatstores BSONdoesnotmake thisrestriction,but
theNode.jsmoduleAPIthataccessesitrequiresit.
Anexample
Let’s pretendyou are runninga business thatsells books overthe internet.
Thefollowingdocumentrepresentswhatyoumightwanttouseinacollection
thatholdscustomersofyouronlinebookstore.Eachcustomerdocumentwould
hold information associated with that customer. You would want to store
information about what books each customer had purchased. The book order
datafor the customer is embeddedin their document. Youwould also wantto
storetheirpersonalcontactinformation.Allofthiscanbeplacedinonesingle
documentasfollows:
//CustomerDocument
{
"_id":"77",
"name":"JoeSchmoe",
"age":27,
"email":"js@live.com",
"address":{
"street":"21MainStreet",
"city":"EmeraldCity",
"state":"KS",
"postalCode":"10021-3100"
},
"booksPurchased":[
{
"title":"AgileProjectManagementwithKanban",
"ISBN10":"0735698953",
"author":"EricBrechner",
"pages":160,
"publicationDate":"20150326",
"category":"SoftwareEngineering"
},
{
"title":"TheMerriam-WebsterDictionary",
"ISBN10":"087779930X",
"author":"Merriam-Webster",
"pages":939,
"publicationDate":"20040701"
"category":"English"
}
]
}
Itisimportanttorememberthatasingledocumentshouldonlycontainthe
informationforthatoneuniquerecord.Thismeansthatyouwouldnotwantto
havetwocustomersinasingledocument.Thatwouldgetconfusing.
Aspreviouslymentioned,documentsdonotneedtobecompletelyuniform.
This means that one document can contain properties that another document
mightnoteverhave,andbothcanexistinthesamecollection.Forexample,if
youhadacollectionthatcontainedproductsforthefictitiousbookstore,itwould
obviouslyhavebooksinit.Itmightalsohavemagazines,maps,andpuzzlesinit
aswell.Thedetailsoftheseproductswouldneedtobesomewhatdifferent.
Documentsinacollectionmighteachhavesomecommonpropertiessuchas
price,title,description,andweight.Foreachproducttype,therewouldbesome
uniqueproperties.Apuzzle,forexample,mighthaveapropertythatstateswhat
therecommendedageisforthatpuzzle.
The following example shows a collection of documents that have both
common and unique properties. Note how the document for a book differs
slightly from the document for a puzzle, yet both can exist in the same
collection.
//Bookstoreproducts
{
"type":"BOOK",
"title":"AgileProjectManagementwithKanban",
"ISBN10":"0735698953",
"author":"EricBrechner",
"pages":160,
"publicationDate":"20150326",
"category":"SoftwareEngineering"
},
{
"type":"BOOK",
"title":"TheMerriam-WebsterDictionary",
"ISBN10":"087779930X",
"author":"Merriam-Webster",
"pages":939,
"publicationDate":"20040701"
"category":"English"
},
{
"type":"PUZZLE",
"title":"Balloonsinsky",
"company":"ZipZapToys",
"age":"3-5yearsold"
}
Just because documents can contain heterogeneous content does not mean
that you always want to have collections set up that way. You may decide to
have all your collections contain uniformly structured documents. I will later
describecircumstancesthatwillhelpyoumakethesetypeofdesigndecisions.
Note:Iwouldadviseyoutostayawayfrommultidimensionalproperties(an
array of arrays). Otherwise, you may find yourself looking through your data
andnotrememberinghowyouhadsetitup.
Acommonpropertyofalldocuments
Each document has an _id property. When a document is created, it will
alwayshaveauniqueidassociatedwithitsothatitcanbeidentified.Thisserves
astheprimarykey.Youcansetthevalueyourself.Ifyoudon’t,MongoDBwill
setavalueforyouoftypeObjectId.Ifyousetityourself,itcanbeofanytype
other than an array. It must, however, be unique across all documents in that
collection.
Ifyoueverneedtodirectlyaccessadocument,youcanaccessadocument
usingthe_idasthequickestwaytoqueryforit.
Referencingexternaldata
Sofar,youhaveseenthatMongoDBallowsforstorageofJSONdocuments.
Asyouknow,thebasicdatatypesavailablewithJSONcanbeusedtorepresent
lotsofinformationyouwishtostore.
Aswasmentioned,theonethingthatJSONdatatypesarenotsuitedfor,is
the representation of large binary data. For example, you would not really be
abletostorelargemediacontentsuchasphotos,music,orvideoinadocument.
Toovercomethislimitation,youcancreateapropertythatisareferenceto
wheretheactualmediacontentisexternallystored.YouwoulduseaUniversal
ResourceIndicator(URI)toindicateitslocation,andcreateaparallelproperty
thatdescribesthetypeofdataitconsistsof:text,image,binarydata,etc.That
way,yourcodecaninterpretitcorrectly.
Theexternallyreferenceddatacanbeanydatayouwouldlike.Itisuptoyou
tosetthetypeandthentreatitassuchwhenyouretrieveit.Besuretohandle
anyneededdeletionofyourexternaldataifitissupposedtobecleanedupwhen
documentsreferencingitaredeleted.
Note:ThischapterstatedthatMongoDB,asadocument-baseddatabase,is
notrequiredtoenforceanyschema.MongoDBdoeshaveacapabilitywhereyou
canspecifywhatpropertiesshouldexistandwhatrestrictionsshouldbeonthem
fordocuments.Idonotutilizethiscapabilityinthisbook.Idolatermentionthat
youcanuseaNode.jsmodulesuchasMongoosetoschematizeyourdataand/or
usejoitovalidateyourschema.ThevalidationMongoDBperformsisalsonot
as strict a definition as you have with a relational database. For example,
relationshipkeyspecificationsareavailablewitharelationaldatabasewiththe
abilitytoalsoenforcewhatisreferredtoasreferentialintegrity.
Chapter2:DataModeling
A data model defines the structure of records that are to be stored in a
database.Thisincludesinformationonhowthedifferenttypesofrecordsrelate
toeachother.
Justbecauseadocument-baseddatabaseisnotarelationaldatabase,doesnot
mean that it doesn’t have structure or relationships between document types.
Specifying a data model in MongoDB consists of specifying what the JSON
documentsare,whichcollectionstheyexistin,andhowtheyrelatetoeachother.
Iwillinstructyouonhowtousevalidationcodetomakesureanydatagoingin
conformstoexpectationsthatconformtoyourdatamodel.
Sketchingout yourdata modelwill be animportant stepin the creationof
yourdata layer. Itis helpful todesign yourdocument structures inadvance so
thatyoucanlookatallaspectsof yourdata.Youcan matchyourdatastorage
needs against the characteristics that a MongoDB database offers and do your
datamodelingaccordingly.
Tovisualizeyourdatamodel,youcanusewhateverdiagramformatortool
youlike.Youcanevenscribbleitoutonapieceofpaper,althoughyoumight
wantotkeepitinelectronicformforeasierediting.Ultimately,withMongoDB,
theJSONformatiswhatyouwouldneedtospecify.Let’snowcoversomeofthe
bigdesigndecisionsthatneedtobemadewithadocument-basedDBMS.
2.1ReferencingorEmbeddingData
Recordsinarelationaldatabaseeachcontainkeystoidentifythemandtoact
asreferencestoeachother.ArelationalDBMShasmechanismstospecifyand
enforcetheintegrityofthesereferencestosomeextent.
You may have heard the term “normalization” used in the context of
relational database designs. Normalization requires the separating out of data
into different record types and relating them to each other. For example, you
mighthavetwotypesofrecordsinyourdatabase,onethatrepresentspeopleand
anotherthatrepresentsthepetsthatareownedbythepeople.Therewouldbea
key to relate an owner to one or more of their pets. The following illustration
showsthisnormalizedstructure:
Figure8-Normalizedstructure
InarelationalDBMS,yousetupthekeystorelatetherecordtypestoone
another.TherelationalDBMSprovidesfunctionalitytobeabletoqueryandjoin
therecordstogether.Thismeansyoucanquerythedatabasetoreturnaperson
andallthepetstheypossess.Thisisdonethroughasinglequeryoperation.
Let’s now bring this conversation into the MongoDB world. Following a
normalized model with MongoDB, you would create separate documents for
eachpersonandeachpetallinthesamecollection.Keepinmindthatthereisno
conceptofaschemaforacollectionrequired,andnoconceptofrelationalkeys
orcross-documentjoinqueries.Youwouldneedtodesignyourownproperties
oneachdocumentinordertorelatepetstotheirowners.Withpeopleandtheir
petsseparatedoutacrossMongoDBdocuments,youhaveachievedanormalized
databasemodel.
Anormalizedmodel,however,isnotnecessarilythebestwayofstoringdata
in a document-based database. You really want to think more about how to
denormalizeyourdata.Denormalizeddatameansthateverythingisallbundled
together.
To denormalize, you bundle pets together with a person. You do this by
simply placing pets as an array property that is embedded in each person
document. Imagine stuffingall the pets in the pocket of the person. This way,
people and their pets are always found together. In this way, there are no
relational links required. You also do not need a join operation if you
denormalizeyour data. In fact, the classic relationaljoin operation is not even
supportedinMongoDB.Thefollowingvisualizationshowshowthepersonand
petstraveltogether:
Figure9-Denormalizedstructure
Aswasmentioned,document-baseddatabaseslikeMongoDBdonotsupport
joinsofrecordssuchascross-tablejoinsdonewitharelationaldatabase.Thisis
becausecross-documentjoinsdonotmakesenseindocument-baseddatabases.
Thismeansthatyouneedtogetcomfortablewithkeepingyourdatabasedesigns
denormalized.Denormalizationismoreefficientandworkswellinadocument-
based database. What you do, is make use of the array and embedded object
propertiesinyourJSON.
In the previous chapter, I showed you a JSON document representing a
customer of an online bookstore company. What was shown was actually a
denormalized data pattern. Each customer document contained information
aboutthecustomer,butitalsocontainedinformationabouteachbooktheyhad
purchased.Youwillendupwithlargerandmorecomplexdocumentswhenyou
denormalizeyourdata.
Even if you end up with large, complex documents in MongoDB, the
queryingcapabilitysupportsretrievalofjusttheportionsofthedocumentsthat
youneed. Forexample, if youkept petsembedded in aperson document, you
candoaquerytojustreturnthepetsofagivenowner.Youdonotneedtoreturn
thecompletepersondocumentifyoudonotwantto.Perhapsyoujustwantto
knowthenamesofallthepetsanddonotwanttoretrieveanythingelseabout
thepetortheperson.
Insummary,itcanbestatedthatdatabasenormalizationhasyouseparateout
yourdataintodifferentdistinctrecordtypesandhasyousetupkeysasreference
linksbetweenthem.On theotherhand,denormalization,with document-based
databases, has you keep as much data bundled together as possible by
embeddingdatathatisrelated.
2.2WhentouseReferencing
Itisobviouslynotgoingtoworkwellifyoualwaysembedallyourdataand
end up with huge, complicated documents. You need to make decisions as to
what properties will be embedded in a document and what would make more
sensetopulloutintoseparatedocumenttypestoreference.
Youcankeepallofyourdifferentdocumenttypesinonecollection.There
are,however,reasonswhydifferentdocumenttypesshouldbekeptinseparate
collectionsorevenseparatedatabases.Iwillcovermoreonthattopiclater.
Performanceimplications
Your data model design will have an impact on the performance of all
database operations. For example, the performance of reads and writes could
improve if you spread data out into separate documents. Doing this, results in
decreaseddatatransferamountsandmoreefficientdocumentupdatesaswell.
The downside of referencing data from one document to another is that it
willbeslowerifyouneedtocombinedataandpresentittogetheratanypointin
time.Imaginethecoderequiredtopiecetogetherapersonwiththeirpetsifpet
documentswerekeptseparate.
Let’s goback to theonline bookstore example.Imagine thatyou have two
documenttypes:customersandbooks.Withthesedocuments,youkeeptrackof
allthebookpurchasesofindividualcustomers.IfyoukeeponlythebookIDsin
the customer document, certain queries would run slower, such as listing a
customerwiththebooktitlestheyordered.Foreachcustomer,youhavetolook
through the booksPurchased array property and then do individual fetches of
dataforeachbookidlisted.Hereishowthisnormalizedmodellooks:
//Customerdocuments
{
"_id":"77",
"type":"CUSTOMER_TYPE",
"name":"JoeSchmoe",
"age":27,
"email":"js@gmail.com",
"address":{
"street":"21MainStreet",
"city":"EmeraldCity",
"state":"KS",
"postalCode":"10021-3100"
},
"booksPurchased":["0735698953","087779930X"]
}
//BookDocuments
{
"_id":"7865",
"type":"BOOK_TYPE",
"ISBN10":"0735698953",
"title":"AgileProjectManagementwithKanban",
"author":"EricBrechner",
"pages":160,
"bookReviews":[
{
"reviewer":"JoeSchmoe",
"comments":"WishIhadthisyearsago!",
"rating":4
},
{
"reviewer":"JaneDoe",
"comments":"Foreveraclassic.",
"rating":5
}
]
}
{
"_id":"7866",
"type":"BOOK_TYPE",
"ISBN10":"087779930X",
"title":"TheMerriam-WebsterDictionary",
"author":"Merriam-Webster",
"pages":939,
"bookReviews":[]
}
Imaginethatyouwanttoqueryandfindallbooksthatareover500pagesin
lengthandthatwerepurchasedbyaparticularcustomer,JoeSchmoe.Todothis,
thequerymustfirstsearchthecollectiontofindtheJoeSchmoedocumentand
then,foreverybookIDlistedintheJoeSchmoedocument,lookupthatbook
documentandseeifthepagespropertyvalueisover500,andthenfinallyreturn
thosedocuments.Thisinvolvesmorethanonedocument.
Inrelationaldatabases,thiscanbedone inasinglequery.InMongoDB, it
mustbedoneinseparatequeriesandinvolvescodetopieceeverythingtogether
ifneeded.
Thisisnotnecessarilyabadthingandisjustpartofwhatyouneedtodoina
document-based database when you separate out data into related documents.
But then again, remember that denormalization relieves you from doing these
joinoperationsacrossdocuments.Youwilllaterseethatthereisacompromise
thatcanbemadebetweenthetwooptions.
Whentoreference
Let’s cover the basic scenarios that would cause you to split data across
documents with a normalized design that uses references. The following are
someofthecommonreasonstodothis:
Thedataisrarelyusedinqueries:
Ifyou find thatyou rarely reference certainproperties in adocument,
thenyoumightseethisasasignthatthosepropertiesbelonginaseparate
document,oreveninaseparatecollectionthatcanbereferencedasneeded.
Forexample,theaddresspropertyinthecustomerdocumentcouldbetaken
out and placed into a separate document if it is decided that it is rarely
needed.Justrememberthatyouaregainingthefasterinteractionwiththe
primary data at the expense of slower data retrieval processing to join it
togetherlater.
Thedataiscommonacrossdocuments:
The book detail is certainly data that would be duplicated across
customers. If you had thousands of customers that each had a Merriam-
WebsterDictionary,itwouldnotbenecessarytokeepduplicatingallofthe
datafor thatbook. Withduplicated data,one bigproblem is ifone ofthe
propertiesofabookneedstobealtered,itwouldrequirealteringitacross
alltheduplicatesinsteadofinonecentraldocument.Inthiscase,itwould
be wise to keep the properties in a separate book document if they are
sharedandupdatedfrequentlyandmakeareferencetothem.
Thedatahasmutualorcross-referencerelationship:
The information in the bookReviews property originates from the
customers that submit the reviews, but each review is also specific to a
singlebook.Thequestionarisesshouldthebookreviewsexistwiththe
book being reviewed, or with the customer giving them? You might not
wanttoupdatetheindividualbook documentandkeepaddingto itevery
time a new customer adds a review. Nor do you want to fetch all of the
bookreviewseverytimeyoufetchthedocumentforabook.Thisiswhere
you can make the case for the book reviews for each book to be in their
ownseparate document withan array property orhave each in individual
documents and then referenced by both the customer and the book
documents.
Thedatawillgrowverylarge:
An array property might grow to have a large number of entries. An
arraypropertyofadocumentcannotgrowunbounded.Rememberthatthere
isa16-megabytelimitonthesizeofanindividualdocumentinMongoDB.
Youwouldhavetosetupseveralrelateddocumentsthateachhadanarray
thatcomprisedsectionsofthetotalset.
2.3ReferenceRelationshipPatterns
AtonepointIshowedyouacustomerdocumentthathadalistthatcontained
theidsofthebooksapersonhadpurchased.Thisdesignkeepsthedocuments
fortheactualbookdetailsseparatedout.Thisisoneofseveralpatternsyoucan
findusefulforreferencingdata.
Iwillstressagainthough,thatthisistotallyunderyourcontroltoimplement.
MongoDB does not provide any built-in recognition of relationships. It is
importanttorealizethatMongoDBwillnotverifythereferentialintegrityofthe
dataasarelationalDBMSmightdo.Itisuptoyoutomanagethat.Ifagiven
bookisdeletedfromacollection,youalsoneedtodeleteallthereferencestoit.
Forexample,ifacustomerdocumenthadabooksPurchasedarraypropertywith
anentryinitwithanidof5777,thereisnothinginMongoDBthatisverifying
thatabookdocumentactuallyexistswithanidof5777.
Herearesomecommonpatternsthatcanbeusedforreferencingdataacross
documentsinMongoDB:
One-to-One: This is where you would separate out a piece of
infrequently accessed data. For example, you could pull out the
address information of a customer and put it into its own document.
Thisdiagramillustratesaone-to-onepattern:
Figure10-One-to-onerelationshippattern
Many-to-One:This is where one document might be referenced
bymanyotherdocuments.Thisistheexampleyouhavealreadyseen
where a single book can be referenced by multiple customers. Each
entryinthebooksPurchasedarrayhasanISBN10IDtoreferencethe
bookdocumentwith.Thisdiagramillustratesamany-to-onepattern:
Figure11-Many-to-onerelationshippattern
Many-to-ManyAssociation:Thisiswhereyouhavetwoseparate
documenttypeswhereneitherreferencestheother,butinsteadthereis
a third document type that can tie the two together through an
association.Thisthirddocumentcanalsocontaininformationrelevant
to that association. For example, you might have a document for
purchasesthatrecordsthedetailsofthetransactionofabookpurchase.
Thus, you have many purchase documents that reference many
customer and book documents. This diagram illustrates a many-to-
manyassociationpattern:
Figure12-Many-to-manyassociationpattern
Many-to-manyrelationshipscouldhavedocumentsreferencingeachotherin
acircularmanner.Trynottogetthatcomplicated,asitpresentsmanydifficult
dataintegrityissuesandcomplicatestheservicelayerimplementation.
Note:Itisusefultocreateavisualdiagramofyourdocumenttypesasyou
gothrough yourdatamodel designprocess. Youarethen ableto moreclearly
see the structure and relationships. Several diagram standards have been
createdovertheyearstovisuallyrepresentobjectsinobject-orientedlanguages
and for representing records in databases. All you really need is a simplified
visual representation of JSON objects in your data model. You can draw a
rectangle shape for each document type and list all the properties inside as I
haveshowninthepreviousillustrations.IhavechosentohavetheIDproperty
existabovealine.Thosepropertiesbelowthelinearetherestofthedocument
properties. I have chosen to use a greater than sign to show sub-object
properties.Anarrayofobjectsisshowninparenthesisanditisunderstoodthere
wouldbezeroormoreoftheseinanactualinstanceofthedocument.Youmight
alsowanttolistthedatatypeofeachpropertyouttotherightofthename.
2.4AHybridApproach
Youare now ready to learn about a hybrid approach thatcan give you the
bestofbothtechniquesforbothreferencingaswellasembeddingdata.Whynot
combinethetwotechniquesinasortofcompromise?Forexample,ifyoufind
thatyouoftenneedtolistthetitlesofthebooksthatacustomerowns,youcan
duplicateaportionofthatinformationacrossdocuments.Thismeansyoustore
thebookidandthetitle,eventhoughitisduplicatedinformation.Thefollowing
isanexampleofwhatthedocumentswouldlooklike:
//CustomerDocument
{
"_id":"77",
"type":"CUSTOMER_TYPE",
"name":"JoeSchmoe",
...
"booksPurchased":[
{
"ISBN10":"0735698953",
"title":"AgileProjectManagementwithKanban"
},
{
"ISBN10":"087779930X",
"title":"TheMerriam-WebsterDictionary"
}
],
...
},
//BookDocument
{
"_id":"77",
"type":"BOOK_TYPE",
"ISBN10":"0735698953",
"title":"AgileProjectManagementwithKanban",
"author":"EricBrechner",
"pages":160,
"publicationDate":"20150326",
"category":"SoftwareEngineering"
},
You can see that the compromise was to keep the title duplicated across
documenttypes,asthetitleisfrequentlyneeded.Therestofthepropertiesare
keptseparatedout.Beawarethatyouwouldwanttosynchronizechangestoany
duplicatedpropertiesthatexisted.Forexample,ifyouneedtomakeachangeto
thetitle property in the book document for a given book, you would want to
havesomebackgroundprocessthatwouldqueryalloccurrencesofthatbookin
thecustomerdocumentsandupdatethetitleinthose.
2.5DifferentiatingDocumentTypes
You might have several different document types existing in a single
collection.Storingmultipledocumenttypesinonesinglecollectionallowsyou
to use a single API connection in your code. API connections are tied to a
databaseandforasinglecollection.
With all document types in the same collection, you would need a way of
differentiatingthemfromeachothersothatqueriescanfindeachspecifictype.
Onewaytosolvethisistoincludeatypepropertyineachdocument.
Let me use the previous example of documents for persons and for pets.
Person documents would be declared with: "type":"PERSON_TYPE" and pet
documentswouldbedeclaredwith"type":"PET_TYPE".Forexample:
//PersonandPettypesinsameCollection
{
"type":"PERSON_TYPE",
"_id":1,
"name":"Ian",
"Petclubmembership":true,
"pets":[12,24],
},
{
"type":"PET_TYPE",
"_id":12,
"name":"Kirby",
"breed":"Cavalierdog",
},
{
"type":"PET_TYPE",
"_id":24,
"name":"Kaitlyn",
"breed":"Siamesecat",
}
Thisallows a query to narrow down results to just returningthe document
typeyouwantandthenyoucanaddinwhateverfurthercriteriayouarelooking
for.
2.6RunningOutofSpaceinaDatabase
Thereare ways to deal withcases where the number or sizeof documents
becomes a problem. Think about what would happen if you keep adding
documents to a collection that existed on a single SSD (Solid State Disk)?
Obviously,atsomepoint,youaregoingtoreachthestoragespacelimitthatis
setforasingledatabaseyouarepayingfor.Thisdoesnotnecessarilymeanthat
youneed to separateout and referencedata across documenttypes inseparate
collections. You can still keep data embedded and grow the number of
documentsindefinitely.
Themethodforachievingstoragecapacityscalingisthroughwhatiscalled
sharding.Thismeansyourcollectionismoreofalogicalconceptandisactually
spread across multiple SSDs. The unit of scaling for MongoDB is called a
replicaset.Agiven document,however,mustonlybefound inonereplicaset
(shard)oftheshardedcluster.ThenicethingisthatMongoDBhidesthatfrom
you,andyourqueryorupdatedoesnotevenrealizewhatisgoingon.
Forcustomerdocuments,youcouldhavethingssetuptodistributecustomer
documents out across different replica set storage. In a later chapter, I will
discussthistypeofdatapartitioning.Don’tworryifyoudonotfullyunderstand
thisconceptjustyet.Youmightneverneedtoimplementshardinganyway,only
ifyourunintoreallylargeamountsofdataandneedtomaintainfastreadand
writetimes.
2.7AccessControl
There are several ways to interact with MongoDB documents. One is
through the Atlas management web portal. Another is through the Compass
application, and another is through API access, such as in Node.js JavaScript
code.Ofcourse,thereisalsotheMongoshell,butthatisnotsomethingthatwill
beneededforthepurposesofwhatthisbookisteaching.Youwilllearnjustthe
bareminimumtousethemongoshell,suchasforimportingdocumentsinbulk.
If you want to learn more about the Mongo shell, please refer to MongoDB’s
documentation.
FromtheAtlasmanagementportal,auseraccountcanbeaddedtoprovide
authenticationforthatuser.Youcangiveauserread-onlyprivileges,ifyouneed
thatrestrictioninplace.Accountadministrationthroughtheportalisnotcentral
to the topic of this book, so if you need more information, please refer to the
Atlasmanagementportaldocumentation.
As mentioned, the primary type of access discussed in this book is done
programmatically through a MongoDB API, such as one that is provided for
Node.jsdevelopers.ThedetailsofthiswillbeexplainedintheNode.jssection
laterinthisbook.
You will later see how the NewsWatcher sample application has a UI that
goes through the middle-tier service layer where interaction with the database
takes place. The middle tier will then be able to authenticate on behalf of the
user and access the database on their behalf and can restrict access to just
documentsthataparticularuserisallowedtosee.
Chapter3:QueryingforDocuments
Youmayhaveheard ofSQL(StructuredQuery Language)asthelanguage
usedtoqueryforrecordsinrelationaldatabases.UsingSQL,yousubmitqueries
to your DBMS and receive back the resulting records that match. MongoDB
supportsquerying,butitdoesnotuseSQL.Instead,ithasitsownquerysyntax.
The lowest level interface to MongoDB has a means of interaction to
accomplish operations such as queries. Everything at the lowest level is done
through a TCP/IP connection that has a well-defined wire protocol for the
operationsthatitsupports.Therearearoundnineoperationsyoucanmakewith
thisprotocol.Don’tworryaboutunderstandingthis,allyouneedtoknowisthat
otherpeoplehavedonetheworktoabstractawaythecomplexitiesofusingthis
protocol by creating specific language drivers you can use. This book will be
concerned with using the Node.js JavaScript driver. The following diagram
showstheoverallaccesslayers:
Figure13-MongoDBaccessabstractionthroughAPIs
In the service layer part of this book, I will show you how to use the
MongoDBNode.jsdriverforoperationsonthedatasuchascreate,read,update,
and delete (CRUD). This current chapter will only cover the specifics of the
syntaxforqueryoperationsingeneral.
Regardless of which of the CRUD operations you perform, you need to
specify your query criteria as part of that request. You can explore this topic
now,becauseyoudon’tneedtowriteanycodetotryoutyourqueries.Youcan
usetheCompassapplicationtotrythemoutandexperimentwiththesyntaxas
youlike.
Exampledocuments
CarefullyreviewtheexampleJSONdocumentsshownbelow.Theywillbe
used with examples showing how to construct your queries. You can imagine
thesedocumentsbeingusedbyanonlinebookstore.Therewouldobviouslybea
lotmoredataavailablethanwhatisinthisexample:
//CustomerDocuments
{
"_id":"77",
"type":"CUSTOMER_TYPE",
"name":"JoeSchmoe",
"age":27,
"email":"js@gmail.com",
"address":{
"street":"21MainStreet",
"city":"EmeraldCity",
"state":"KS",
"postalCode":"10021-3100"
},
"rewardsPoints":99,
"booksPurchased":[
{
"id":"1098",
"title":"AgileProjectManagementwithKanban"
},
{
"id":"1099",
"title":"TheMerriam-WebsterDictionary"
}
]
},{
"_id":"78",
"type":"CUSTOMER_TYPE",
"name":"JaneDoe",
"age":37,
"email":"jd@gmail.com",
"address":{
"street":"100SBridgerBlvd",
"city":"Paradise",
"state":"UT",
"postalCode":"84328"
},
"rewardsPoints":0
}
//BookDocuments
{
"_id":"1098",
"type":"BOOK_TYPE",
"title":"AgileProjectManagementwithKanban",
"ISBN10":"0735698953",
"author":"EricBrechner",
"pages":160,
"format":"Paperback",
"price":27.66,
"publicationDate":"20150326",
"category":"SoftwareEngineering",
"bookReviews":[
{
"reviewer":"JoeSchmoe",
"date":"20140321",
"comments":"WishIhadthisyearsago!",
"rating":4
},{
"reviewer":"JaneDoe",
"date":"20150923",
"comments":"Foreveraclassic.",
"rating":5
}]
},{
"_id":"1099",
"type":"BOOK_TYPE",
"title":"TheMerriam-WebsterDictionary",
"ISBN10":"087779930X",
"author":"Merriam-Webster",
"pages":939,
"format":"Paperback",
"price":11.26,
"publicationDate":"20040701",
"category":"English",
"bookReviews":[
{
"reviewer":"JoeSchmoe",
"date":"20100101",
"comments":"Aterrificvolumetokeephandy.",
"rating":4
},{
"reviewer":"JaneDoe",
"date":"20120817",
"comments":"Wishitcameinanaudiobookformat.",
"rating":2
}]
}
Iwillnowintroduceyoutothebasicsofthequerysyntax.Iwon’tcovereach
andeveryaspectofit.Itisfairlyrobustandmuchofitisbeyondthescopeof
this book and something you might not ever need to use. For example, I will
only briefly mention how to use the MongoDB aggregation features and its
syntax. For more information on that topic and other supported syntax
intricacies,refertoMongoDB’sdocumentation.
Syntaxoverview
Inalaterchapter,youwillbequeryingacollectionusingaMongoDBNPM
moduletowritecodeinJavaScriptthatrunsinNode.js.Thatmoduleprovides
functions such as find() or findOneAndDelete(). The first parameter of those
functionsisthequerycriteriathatspecifiesthematchingtotakeplaceacrossall
ofthedocumentsinacollection.
IfyouarefamiliarwithSQL,thisissimilartowhataWHEREclausedoes.
Hereisanexamplequeryusingthefind()functionwithagreaterthanoperatorin
the query criteria. This query below will return all of the documents in the
collectionwhoseagepropertycontainsavaluegreaterthan35.Thisisshowing
youhowyouwouldcallitincode,andthisalsohappenstobetheformatyou
useintheMongoshell.
db.collection.find({age:{$gt:35}});
For the function call above, there is the possibility that the query criteria
won’t match anything. This is ok, and no documents will be returned. On the
other hand, if there are a lot of documents returned, then you need to use the
Node.jsdrivercapabilitytofetchresultsinbatches.Youwillseehowtoactually
accesstheresultsofthefind()functionincode.
BesidesthequerycriteriaIjustshowedyou,therearealsocriteriayoucan
provide for what is called the projection criteria. The projection criteria
determine the properties that will be returned from each document. Here is
anotherexampleusing the samefind()function, butthistime withanoptional
secondparameterthatspecifiestheprojection:
db.collection.find({age:{$gt:35}},{name:1,age:1});
This query will find all documents in the collection whose age property
containsavalue greaterthan 35.Withthe projectionspecified, onlythename,
age and _id properties will be returned. _id is always returned, unless you
specifyotherwise.
Asmentioned,thesecondparameterin thisqueryisthe projectioncriteria.
This is where you list the properties you want to be returned. The number
followingthecolondeterminesifthepropertyisincluded,orifitisexcluded.I
willexplainmoreaboutthissoon.
Iwillkeepusingtheexampleofthebookstorecustomerdocument,butfor
now, just pretend they only have four properties each (_id, name, age, and
email).The followingdiagram of theprevious query showsthe two criteriain
thefunctioncallandhowtheydeterminetheoutput:
Figure14-Criteriaflow
Youcanseethatthequerycriteriadeterminewhatdocumentspassthroughto
the result set. The projection criteria select what properties you want for each
documentintheresultset.
Now I can go into the details of both the query and the projection criteria
operations. You can connect to your MongoDB hosted cluster through the
Compassapplication,addadatabase,addacollectionandsomedocuments,and
thentryoutsomequeriesonyourown.
Note:You cannotjust create a query and assumethat it will endup being
efficient.Theexecutiontimeofaquerycanvarygreatly.Toaddressperformance
issues, you either must create indexes that can speed up your queries or think
aboutamoreefficientwayofmodelingyourdata.Thetopicofindexcreationis
coveredlater.
3.1QueryCriteria
The query criteria are actually optional on a function such as the find()
function.Itis,however,somethingyouwillalmostalwaysbeusing.Ifyoucall
find() without any query criteria parameter, every single document in the
collectionwillbereturned.
The query criteria are tests that are applied to the collection to see what
documentsaretobeincludedaspartoftheresultset.Ifyoureallywantto,you
cannarrowdowntheresulttoreturnasingledocument.Forexample,youcan
querybythe_idpropertywithan equalitytest.The _idpropertyis unique,so
eachdocumentcanbeuniquelyidentifiedwithit.Hereisanexampleofcodeto
queryby_id.Theresultitreturnsisalsoshown:
//Query
db.collection.find({_id:{$eq:"77"}},{address.state:1});
//Results
{
"_id":"77",
"address":{
"state":"KS"
}
}
Youdon’tneed to writeany code totry this out, butcan use the Compass
applicationtotryoutqueries.Youcangotochapter6tolearnhowtocreateyour
PaaS hosted MongoDB cluster, database, and collection. You will also find
informationoninstallingthemongoshell.Youcanusethatknowledgetoimport
documentstouseasyouexperimentwithqueries.SeetheExampleDocuments
section,afewpagesprevious,forthedocumentstocreate.
Onceyouhaveyourtestdatabase,testcollection,withdocumentsimported,
you can launch Compass and go to your collection and in the Documents tab,
entersometestqueries.HereiswhattheCompassUIlookslikeifyouaretrying
outaquery:
Figure15-Compassapplicationqueryingcapability
Each query criteria can utilize one or more operators. The example above
usesthe$eqoperator.The real powerof the query is inthe use of thecriteria
operators.I’llnowgooverwhatthoseareandshowyousomeexamples.
Criteriaoperators
Youhave seen operators such as $gt and $eqused in the examples in this
chapter. Those operators stand for greater than and equal to. There are many
moreoperatorsthatyoucanuseinyourquerycriteriatofilterdocuments.The
followingoperatorsarecurrentlysupported:
Comparison
$eq
$gt
$gte
$in
$lt
$lte
$ne
$nin
Array
$all
$elemMatch
$size
Bitwise
$bitsAllClear
$bitsAllSet
$bitsAnyClear
$bitsAnySet
Element
$exists
$type
Evaluation
$expr
$jsonSchema
$mod
$regex
$text
$where
Geospatial
$geoIntersects
$geoWithin
$near
$nearSphere
Logical
$and
$or
$nor
$not
Eachoftheoperatorsusesitsownuniquesyntax.The$eqoperatorusesthe
followingsyntax:
{<name>:{$eq:<value>}}
Thenameiswhatyouwanttotestagainst.Itcanbeatop-levelproperty,orit
can be a property within the hierarchy of the JSON. The value is a string,
number or other value that matches the data type of the property. Here is an
examplethattestsasecond-levelproperty;referredtoasanembeddeddocument
fieldinMongoDBdocumentation:
{"address.state":{"$eq":"UT"}}
Youcanevenspecifyapropertyofanelementfoundinanarray.Ifyoulook
at the example document, you see that booksPurchasedis an array and idis a
propertyofeachoftheelementsofthatarray.Intheexamplebelow,thereturned
documentisthecompletedocument,asthequerycriteriaareonlyusedtofinda
documentmatchandnotspecifypropertiestoreturn.
{"booksPurchased.id":{"$eq":"1098"}}
Hereareafewoperatorsfromsomeofthecategoriestogiveyouanideaof
how they work. For more detailed information on each of the operators, see
MongoDB’sdocumentation.
Comparison
With the comparison operators, you need to first select the property name
you are interested in testing against. This is then followed by the comparison
operator and finally the value you want for that comparison test. The only
exceptiontothisiswiththe$inand$ninoperators,whichuseanarrayandnota
singlevalue.
IwillusethesameexampleIhaveshownyoupreviously,whichisdoinga
queryforasingledocumentbyits_idvalue:
{"_id":{"$eq":"77"}}
To just test the equality of a property, you can shorten the syntax to the
following:
{"_id":"77"}
Youcanusemorethanonecomparisonoperatoratatime,suchasyouwould
needtodototestrangesof values.Alloperatortestsneedtopasstheir testin
orderforagivendocumenttobeincludedintheresultsset.
Here is an example that queries for books that have between 100 and 200
pages:
//Query
{
"pages":{
"$gt":100,
"$lt":200
}
}
Logical
Thelogicaloperatorsletyoustringtogetherseveraltestsinarowtoperform
thedesiredlogicaltesting.Thesyntaxforthelogicaloperator$andis:
{$and:[{<expression1>},{<expression2>},...,{<expressionN>}]}
The logical operator syntax starts with a boolean operator such as $and.It
then contains an array of expressions that can be made up of individual
comparisonoperatorsthatwehaveseenpreviously.
Thefollowingquerylooksforbooksthatarelessthan200pagesinlength
andwhicharealsointheSoftwareEngineeringcategory.
//Query.
{"$and":[{"pages":{"$lt":200}},
{"category":{"$eq":"SoftwareEngineering"}}]}
//Results.Assumingyoualsohaveaprojectioncriteriaof{"_id":1}
{
"_id":"1098"
}
If you are only carrying out this one level of boolean operation, then you
don’treallyneedthe$andoperator.Instead,youcanjustlisttheconditionsone
afteranother.Hereisthesameexamplequerywithoutthe$andoperator:
//Query.
{"pages":{"$lt":200},"category":{"$eq":"SoftwareEngineering"}}
//Results.Assumingyoualsohaveaprojectioncriteriaof{"_id":1}
{
_id":"1098"
}
Here is a query that uses both the $and and the $or operators. I’ll use the
complete expanded text formatting of the query as it is easier to read. This
example queries for books that have less than 200 pages and are either in the
categoryofSoftwareEngineeringorScienceFiction.
//Query.
{
"$and":[
{
"pages":{
"$lt":200
}
},
{
"$or":[
{
"category":{
"$eq":"SoftwareEngineering"
}
},
{
"category":{
"$eq":"ScienceFiction"
}
}
]
}
]
}
//Results.Assumingyoualsohaveaprojectioncriteriaof{"_id":1}
{
"_id":"1098"
}
Ifyoufindyourqueryhasalotof$oroperationstomatchonmanydifferent
values for the same property, then you can use the $in operator. The value to
matchcanevenbearegularexpression.Thefollowingexampleshowshoweasy
itistousethistolistallthepossiblematches:
//QueryusingIN
{
"category":{
"$in":[
"SoftwareEngineering",
"ScienceFiction"
]
}
}
Element
Theelementoperators$existsand$typeareforselectionsbasedonwhether
apropertyexistsandifitisofacertaindatatype.Thefollowingexampleshows
thesyntaxfor$exists:
{name:{$exists:<boolean>}}
A typical use for $exists would be to use this operator inside another
operator. In this example, you want to make sure the document has the
publicationDateproperty.Itmaybethatabookhasapriceset,buthasnotbeen
publishedyet,sothepublicationDatepropertyisnotthereyet.
{"$and":[{"price":{"$lt":30}},{"publicationDate":{"$exists":true}}]}
Evaluation
Ifyoustruggletogetexactlywhatyouwantinyourquery,youmightfind
theevaluationoperatorsarejustwhatyouneed.Hereisanexamplethatshows
theuseof aregular expressionwiththe optionfora case-insensitivetest.This
examplewillfindallbooksthathaveatitlethatstartswiththeword“agile”no
matterthelettercasing:
{
"title":{
"$regex":"^agile",
"$options":"i"
}
}
You might have a document with large amounts of text that you want to
searchtoseeifspecificwordsorphrasesexist.Youcanusethe$textoperatorin
this case. To use this, you need to first create an index of type text on the
propertiesyouwanttouseiton.Forexample,youcouldcreatetheindexonthe
titlepropertyandthensearchforbooksthatcontaincertainwordsintheirtitle.
{"$text":{"$search":"MongoDB"}}
Forthoserareoccasionswhereyoujustcannotgetwhatyouwantwithany
oftheavailableoperators,youcanresorttowritingJavaScriptusingthe$where
operator.Hereisatestthatcheckstoseeifapersonhaspurchasedmorethana
single book. This requires JavaScript because the length property of the
booksPurchasedarrayisonlyaccessiblethroughtheAPIreturnedobject.
{"$where":"this.booksPurchased.length>1"}
YoucannotpresentlytrythisqueryoutfromtheCompassappifyourhosting
issetuptousetheAtlasfreetier.Thesameistrueforyourcode.
Objectandarrayproperties
So far, the examples shown have been testing properties that are single
values,suchasastringoranumericdatatype.Butwhatifyouhaveadocument
that has a property that is an array of strings? What if you have an object
property?Evenbetter,whatifyouhaveapropertythatisanarrayofobjects?
The address object property in the bookstore customer example document
endsupasanembeddeddocumentinMongoDBBSON.Youcandoasearchfor
an exact match on an embedded document and specify individual names to
matchasshowninapreviousexample.
Forarrays,youcandoanexactmatchonthefullcontentsofthearray,orjust
on specific values existing somewhere within the array. If the array holds
objects,youcansearchfortheelemententryandsub-propertyoffofthat.Here
isapreviousexamplequerythatwasdoingthis:
{"booksPurchased.id":{"$eq":"1098"}}
Justasyoucantestmultiplesinglepropertyvalues,youcanalsodothatfor
propertiesthatarearraysobjects.Let’ssayyouwantedtosearchforbookswith
bookreviewsbyJoewherehegaveafour-starrating.Hereanexampleofhow
thatwouldlook:
{
"bookReviews.reviewer":"JoeSchmoe",
"bookReviews.rating":4
}
Forasimplearrayofstrings,youcouldmatchforthatexactarray.Tosearch
fordocumentswhereonestringentryinanarrayexists,youcoulddoanequality
test.Hereisanexampledocumentwithapropertythatisanarrayofstrings:
{
"favoriteColors":["green","red","blue"]
}
Toincludethatdocument,herearethequerycriteriayoucoulduse:
{"favoriteColors":"green"}
ArraysearchesandprojectioncapabilitiesinMongoDBareverypowerful.If
you take the time, you can learn how to match on things like an element in a
specificindex,ordosomethinglikereturnthefirstnumericelementthatislarger
thansomevalue.Youwillhavetolearnthatyourself,asitistrickytoexplainall
the nuances. See the MongoDB documentation. For example, look at the
$elemMatchoperatordocumentation.
Datatypemismatchproblem
Equality comparisons can end up producing an undefined outcome if the
propertydatatypespecifieddoesnotmatchupwiththetestvaluedatatype.For
example,youcannottestforanumbervalueonapropertythatisastring.
TheteststatementsyntaxofMongoDBdoesnotworkthesameasitdoesin
theJavaScriptlanguage.Thefollowingquerywillnotworkbecauseofthedata
typemismatch:
//Theselectionwillnotwork,asthe_idpropertyisastring
//inthedocument,andyouarecomparingitwithanumber
{
"_id":{
"$eq":77
}
}
With the following JavaScript code sample, you can see that data type
coercionhappens.Abooleantestbetweenastringandanumberactuallyworks
inJavaScript.
//JavaScriptuses"=="forequalitytesting.
//Coercionrulesapplyandbothequalitytestsevaluatetotrue
varv="77";
v=="77";//trueresult
v==77;//trueresultascoercionhappens
3.2Projectioncriteria
Justbecauseyouhaveyour querycriteriareturningtheproperresult setof
documents does not mean that you are done. You may also want to set up
projection criteria to just return the properties that you really need. You have
seenthisdemonstratedalready,butnowyoucanlookatthisinmoredetail.
There may be cases where documents with all of their properties are what
youactuallywantreturned.Thismaybethecasewithaverysparsedocument,
makingitreasonabletoreturnthewholedocumenteverytime.Withlarger,more
complex documents, you can benefit from restricting the properties being
returned. Limiting what properties are returned saves on the amount of data
transferred.
Inclusionandexclusion
Youhavealreadyseenaprojectioncriteriainuse,soyoureallyknowmost
ofwhatyouneedtoknowalready.Justtoreview,ifyoudon’tprovideprojection
criteria, then the complete document is returned. If you do provide projection
criteria,thenyoucanspecifytheinclusionorexclusionofwhicheverproperties
youwouldlike.Exclusionmeanstoreturnallpropertiesexcepttheonesyoulist.
Theinclusionandexclusionsyntaxisasfollows:
<name>:<1ortrueor0orfalse>
True means to include and false means to exclude. You cannot mix both
inclusion and exclusion in the same projection criteria. The only exception to
this is if you are using inclusion criteria, you can also specify one single
exclusionifitistoexcludethe_idproperty.
Here are some examples of different projection criteria with a comment
addedtostatewhethertheyarevalidorinvalid:
{"address.state":1}//Valid
{"address.state":0}//Valid
{"name":1,"age":1}//Valid
{"name":1,"age":0}//Invalid
{"age":0}//Valid
{"name":1,"_id":0}//Valid
AsImentioned,thisisextremelyhandy.Let’ssayyouwanttocreatealistof
peoplewiththeiraddresses.Youcouldusethefollowingprojectioncriteria:
{"name":1,"address":1,"_id":0}
Thefollowingistheresultsetreturned:
{
"name":"JoeSchmoe",
"address":{
"street":"21MainStreet",
"city":"EmeraldCity",
"state":"KS",
"postalCode":"10021-3100"
}
}
{
"name":"JaneDoe",
"address":{
"street":"100SBridgerBlvd",
"city":"Paradise",
"state":"UT",
"postalCode":"84328"
}
}
Missingproperties
Since MongoDB can be schema-less, it is possible that any number of
documentsinacollectionthatyouarequeryingmightnotevencontainthegiven
property that you have specified in your selection criteria. For example, it is
possiblethatbooksPurchasedisamissingpropertyinsomeofyourdocuments,
by your own design. This is important to consider when you are constructing
yourquerycriteriaandprojectioncriteria.
The following example query will return two documents, but the second
documentwillnothavethebooksPurchasedproperty.Thisisbecausethesecond
customerhasnotboughtanybooksyet.
//Projectioncriteria
{"name":1,"booksPurchased":1}
//Results
{
"_id":"77",
"name":"JoeSchmoe",
"booksPurchased":[
{
"id":"1098",
"title":"AgileProjectManagementwithKanban"
},
{
"id":"1099",
"title":"TheMerriam-WebsterDictionary"
}
]
}
{
"_id":"78",
"name":"JaneDoe"
}
Ifyoureallywantthisseconddocumentleftoutcompletely,iftheproperty
does not exist, use a query selector to only get those with a non-null value as
shownhere:
//Querycriteria
{
"booksPurchased":{
"$ne":null
}
}
Ofcourse,thepropertycouldstillexistandjustbeazero-lengtharrayandit
wouldbereturnedinthatcase.
Arrays
There is a special operator named $slice that allows you to return just
specificportionsofarrayproperties.Examinethefollowingdocumentthathasa
propertycontaininganarrayofcolors:
{
"favoriteColors":["green","red","blue"]
}
Hereareafewexamplesoftheuseofdifferentoperatorslike$slice,$,and
$elemMatchtopulloutdifferentelementsfromthearraypropertyshownabove:
//Returnfirsttwoelements
{favoriteColors:{$slice:2}}
//Returnfirstelement
{favoriteColors.$:1}
//Returnfirstelementthatmatches
{favoriteColors:{$elemMatch:{$eq:"red"}}}
Formoreinformationontheseoperators,seeMongoDB’sdocumentation.
3.3QueryingPolymorphicDocumentsina
SingleCollection
In the section on data modeling, I mentioned that you may decide to
normalizesomeofyourdata.Youmightliketostoredataincompletelydifferent
document types in the same collection. To do this, your query must always
includeawaytopickoutjustthedocumentsofaparticulartypethatyouwantto
havereturned.
Using the bookstore example, if you had the customer and the book
documentsinthesamecollection,youcouldincludeatypepropertyineach.
Thefollowingisanabbreviatedexampleshowingthisapproach:
//Customerdocuments
{
"_id":"77",
"type":"CUSTOMER_TYPE",
"name":"JoeSchmoe",
...
}
{
"_id":"78",
"type":"CUSTOMER_TYPE",
"name":"JaneDoe",
...
}
//BookDocuments
{
"_id":"1098",
"type":"BOOK_TYPE",
"title":"AgileProjectManagementwithKanban",
...
}
{
"_id":"1099",
"type":"BOOK_TYPE",
"title":"TheMerriam-WebsterDictionary",
...
}
Ineveryquery,youwouldneedtoincludequerycriteriathatwerespecificto
thetypeofdocumentyouneeded.Hereisanexampleofthatquerycriteria:
//Querycriteriaforretrievingallbooks
{"type":"BOOK_TYPE"}
Chapter4:UpdatingDocuments
The previous chapter covered the topic of querying or the ‘R’ for Read
operationsintheCRUDacronym.Thischapterwillcoverthe‘U’fortheUpdate
operation.Iwillgiveanoverviewhereofhowthesyntaxworks.
When updating a document, you can certainly provide the complete
documentforuploading.Asanenhancement,MongoDBallowsyoutodothings
likespecifyingasinglepropertytobeupdated.Therearemanyupdateoperators
you can choose from and they can be combined in a single atomic update
submissionforagivendocument.
To use the update capability of MongoDB, you first need to provide the
querycriteriatoidentifythedocumentordocumentstobeupdated.Thatcriteria
usesthesamesyntaxasalreadycoveredforthequerycriteria.Whatisnewhere
is that the update criteria is added as another parameter. Here is the update
criteriasyntax:
{
<operator1>:{<name1>:<value1>,...},
<operator2>:{<name2>:<value2>,...},
...
}
Note: Create and Delete operations are being skipped. Creation is just
providingtheJSONdocumenttocreateandadeletionoperationusesthesame
querycriteriasyntaxthatareaddoes.AllMongoDBCRUDoperationswillbe
coveredintheservicelayerdiscussion.
4.1Updateoperators
Thefollowingarethecurrentlysupportedupdateoperationsusedonsingle
valueproperties:
$currentDate
$inc
$max
$min
$mul
$rename
$set
$setOnInsert
$unset
Herearesomeexamplestoillustratesomeoftheseupdateoperators.Iwill
only show examples that update a single property at a time. Youcan combine
multipleoperatorsondifferentpropertiesinoneupdatesubmission.Thesewill
allbecommittedatthesametimeandwillresultinanatomicoperationatthe
documentlevel.
Therearemanyoperatorsyoucanuse,andIwillgiveexamplesofafewof
them.Foreveryupdatecall,youneedasthefirstparameterthequerycriteriato
identifythedocument(s).Thesecondparameterspecifiesthepropertytoupdate.
Ifthequerycriteriaidentifymorethanonedocument,theupdatehappensonall
ofthosedocumentsidentified.
Note: The Compass app does not allow the use of the update syntax right
now, only query searches work. That is why I am showing the examples with
code.
$set
The$setoperatoristhestandardwaytoreplacethevalueofaproperty.The
followingexamplesetstherewardsPointspropertytoanewvalueforthequeried
person:
db.collection.update({_id:"77"},{$set:{rewardsPoints:1000}});
Ifthepropertydidnotexistinthedocument,itiscreated.Thisoperatorcan
beusedtodoacompletereplacementofanyvalue,evendoingareplacementof
acompletearrayoranembeddeddocument.Italsoworkstoreplaceaspecific
propertyofanembeddedobject.
$renamecan be used to give a property a new name. $unset will delete a
property.
$inc
The $inc operator is used to change the integer value of a property by a
specifiedamount.Youcanaddorsubtractfromanyvalue.Usingthebookstore
example, you could add rewards points to a customer. Here is an example of
whatthatwouldlooklike:
db.collection.update({_id:"77"},{$inc:{rewardsPoints:10}});
$minand$max
The$minand$maxoperatorsareusedtotestagivenvalueandonlyreplace
it if the value is less than or greater than the test value, depending on the
operator.Hereisanexample:
db.collection.update({_id:"77"},{$max:{rewardsPoints:2000}});
In this example, if the rewardsPoints property had a value of 1000 to start
with,ithasavalueof2000afterthisupdate.
4.2ArrayUpdateOperators
Thefollowingoperatorsareforusewitharraypropertiestoperformupdates:
$
$[]
$[<identifier>]
$addToSet
$pop
$pull
$pullAll
$push
$push
The$pushoperatorisusedtoaddanotherelementtoanarrayproperty.Here
isanexamplethataddsanewbookreviewtoabookdocument:
db.collection.update({_id:"1098"},
{$push:{bookReviews:{
reviewer:"Skylar",
date:"20150923",
comments:"Itwasreallyprofound!",
rating:5
}}});
The $addToSet operator is similar to $push except it checks to see if an
identicalentryexistsalreadyandonlyaddsthenewelementifitisnotalready
presentinthearray.Youcanusetheadditionaloperatorof$eachtoaddmultiple
elementsatonce.The$sortoperatorcanbecombinedwiththe$pushand$each
operatortokeepthearraysorted.Combinethe$positionoperatorwith$pushto
specifythepointofinsertion.
$
The$ operator is used to specify that the update is to happen for only the
firstelementofanarraythatisfoundtomatchthequerycriteria.Thepartofthe
$setthat uses bookReviews.$.rating uses the .$to signifythat thereplacement
should take place on just the first element match that is found. This example
doesasearchforanydocumentthathasabookReviewselementthathasarating
of4andthenupdatesthefirstmatchedelement:
db.collection.update({_id:"1098",bookReviews.rating:4},
{$set:{"bookReviews.$.rating":5}}
)
$pop
The$popoperatorisusedtoremoveelementsfromanarrayproperty.This
exampleremovesthefirstbookreview:
db.collection.update({_id:"1098"},{$pop:{bookReviews:-1}});
Youcanuse$pulltoremoveallentriesfromanarraythatmatchwhatyou
specify.Youcan use$pullAll to specify more than one match forthe removal
criteria.
4.3Transactions
A single write operation might modify multiple documents if the selection
query matched more than one document. The modification of each individual
documentisatomic,butitisnotatomicacrossallofthemodifieddocuments.It
is possible to isolate a single write operation across multiple documents using
the$isolatedoperator,butonlyifthereisnoshardinginvolved.
Youmightalso have aneed to modify twodocuments simultaneously, and
evenmodifydifferentpropertiesoneach. Ifyouhavearequirement tochange
twodocumentssimultaneously,thenyouneedtoworkthisoutonyourown.For
example,ifyouwantedtotakerewardspointsfromonedocumentpropertyand
add them to a different document, this would require what is termed a multi-
documentatomictransaction.
This is a new capability in MongoDB 4.x. Previously, you would need to
build that capability yourself. Search “two-phase commit” as per MongoDB’s
documentation.
Youcanimaginehowimportantthiswouldbetogetrightinanapplication
that manages financial transactions across financial accounts. MongoDB now
supportsmulti-documentACIDtransactions.Youshouldonlyturnthisonifyou
reallyneedit.Thepointisthatadocument-baseddatabasemightnotneedthisin
mostcircumstances.Inthecode,youwoulduseanAPIcalltostartasessionthat
willsurroundseveraloperationsthatyouwanttobetransacted.Thenyouwould
eitherabortitorcommitit.Basically,somethinglikethis:
s=client.start_session()
s.start_transaction()
…codetodomultipleupdates,insertsetc.ondifferentdocuments.
--->onanerrorexceptionyouwouldcalls.abort_transaction()
--->onsuccessyouwouldcalls.commit_transaction()
s.end_session()
Chapter5:ManagingAvailabilityand
Performance
Inanidealworld,youcouldstoreaninfiniteamountofdata,accessitfrom
anywhereinnear-zerotimeandneverhaveanydatalossorcorruption.Realityis
that it takes a lot of work to approach these ideals. As you design your data
modelandsubsequentlytryitout,youneedtotuneyourDBMSforconsistency,
availability, and performance. You can now consider what mechanisms are at
yourdisposaltoapproachtheseideals.
It really takes a fair amount of time to fine-tune each aspect of the
managementof your MongoDB database. Many times, you will be faced with
tradeoffsthathavetobemade.Thischapterwilllookatsomeoftheaspectsthat
canbe“fine-tuned”forspecificaccessscenarios.
In a PaaS environment, some of this work should be less than has been
traditionallyrequiredinthepastwithaDBMS.MongoDBhascertainlydonea
greatjobofmakingsomethingsautomaticallyhappenthatusedtorequirealot
ofmanualconfiguration.
5.1Indexing
Imagine you had a problem finding personal belongings such as your car
keys, the TV remote, your favorite pair of socks, your wallet or purse, etc.
Perhaps when you try to head out the door, you find yourself frantically
searchingforyourcarkeyseverymorning.
One approach to solving this would be to keep a whiteboard right next to
yourfrontdoorthathadtwocolumns.Onecolumnwouldlisttheitemyoucared
about and the other column would list the known location of that item. The
whiteboardmightlooklikethis:
Carkeys
Left side pocket of the jacket hanging in the entryway
closet
TVremote Inthepileoftoysintheroomofyourtwo-year-oldtoddler
Purple
socks
Laundryroomfloorunderthepileoftowels
Wallet UnderthecouchcushionintheTVroom
Imaginethehugetimesavingsthiscouldprovide.Isawonestudythatstated
that,onaverage,apersonspendsawholeyearofaccumulatedtimelookingfor
lostitemsduringtheirlifetime.
A database index uses the same concept as the whiteboard look-up table,
with the goal of serving database records faster. A database index works by
creatingaseparatelookuplistthatallowsforfasterquerying.Thisalleviatesthe
needtosearchthroughalldocumentstofindtheone(s)youarelookingfor.For
example, let’s say you had a lastName property in every document of a
collection.Ifyoucreatedaquerythatwaslookingforaparticularpersonwith
thelastnameof“Smith”,howwouldaqueryfinditasquicklyaspossible?The
slowestwaytosearchwouldbetostartlookingatallthedocumentsonebyone,
untiladocumentwasfoundwith“Smith”inthelastNameproperty.Thattypeof
search has no choice but to search each and every document in an unsorted
storage system. In a huge collection, this would be a major performance
problem.
Indexingcanspeedupyoursearchbycreatingaseparatesortedlistoflast
namestosearchagainst.Eachentrywouldpointtothecorrespondingcomplete
document.A queryfor thelastname ofSmithquickly findsthose entrieswith
somethinglikeabinarysearch.
Asanexample,hereisarepresentationofrandomdocuments.Thereisone
rowperdocumentthatexistsinaMongoDBcollection.Thisisonlyanabstract
representation of how it is stored. To search for “Tuttle,” you would start a
sequential search from document to document until you found a match on the
lastName.Unfortunately,inthiscase,itwouldbethelastonefound.
Figure16-Customerdocumentsunsortedandwithnoindex
IfyouaddasecondlistthatcontainsthelastNameproperty.Alongwiththe
name,youcouldhavealinktothecustomerdocument.Thissecondlistcanbe
sorted, and your searching will be much faster. A quick binary search of the
index list will find the lastName to match and then use the link to get the
document.
Implementingagoodindexcanbeacriticalpartofyourworktomaximize
theefficiencyofyourqueries.Itiswellworthyourtimetomeasureandanalyze
the performance of your database and to fine-tune it with indexes. There are
reportsin thepaid Atlastier subscriptionsthat youcan bring upthat helpyou
lookatindexperformance.
Thefollowingexampleillustratestheconceptoflinkingthroughanindex.
Figure17–LastnameindexforCustomerDocumentquerying
YoucanusetheCompasstoolforcreatingallyourindexes.Thereareways
todothisprogrammatically,butforareasoftheapplicationthatonlyneedtobe
setuponce,IalwaysprefertodothisinCompass.
Note:IndexconfigurationcangetrathercomplexandIcanonlycoverthe
basiccommonscenariospertainingtothesampleapplication.Youwillhaveto
gototheonlinedocumentationtogetallthedetailsonwhatispossible.
Single-propertyindex
Thesimplestwaytolearnaboutindexesistolearnhowtosetupanindexon
a single property. This section goes over how this works for a single value or
object property. The next topic explains the subtleties of what happens if the
propertyisanarraydatatype.
Let’s go back to the example of the online bookstore. If you look at the
requirementsforyourquerying,youcanseethatyouneedtobeabletosearch
for customers by name. If you had hundreds of thousands of customers, it is
certainlygoingtoimprovetheperformanceofthisqueryifyoucreateanindex
onthenameproperty.
Thesyntaxusedtocreateanindexissimilartothesyntaxusedtosetupyour
othercriteria.Thisexampleshowshowtospecifyanindexonname:
{"name":1}
That is how simple it is. You can also add an index on a property of an
embeddeddocument.Forexample,whatifyouwantedtolookupallcustomers
thatresidedinacertainpostalcode?Tomakethisqueryrunfaster,youwould
wanttoplaceanindexonthesub-propertyasfollows:
{"address.postalCode":1}
Youcouldplaceanindexontheaddresspropertyasawhole,butthenyou
wouldhavetoputinacompleteaddressforthequerycriteria,includinghaving
thepropertiesinthesameorderforthematchtosucceed.
Note: The _id property that exists on every document is automatically
indexed,soyouneverneedtoaddanindexforthat.
Arraypropertyindex
Ifadocumentpropertyisanarray,yousetupanindexonitinthesameway
asforasinglevalueproperty.Therearejustafewrestrictions.Onerestrictionis
thatyoucannothavemorethanonearraypropertyindexatatime.Forexample,
ifyourdocumentshavemultiplearrayproperties,youcanonlysetupanindex
ononeofthearrayproperties.
Theindexwouldthenbeusedtomatchonanythingfoundinthearray.The
indexcanonlybeusedforindividualelementmatchingandnotformatchingon
thearrayasawhole.Asanexample,let’ssayyouhadthefollowingdocuments:
{"name":"Kiara","favoriteColors":["blue","yellow","cyan"}
{"name":"Tristan","favoriteColors":["green","red","blue"}
{"name":"Halli","favoriteColors":["juju","nana","mango"}
Youcansetupanindexsimilartothepreviousexample:
{"favoriteColors":1}
Nowyoucansetupaquerywithquerycriteriatomatchacolorthatmight
befoundinthearray.Ifyousearchfor“red,”thenthedocumentforTristanwill
be included in the results set. What is happening internally is that the index
contains all array element entries across all documents. There would be three
entriesintheindexforKiara,alphabeticallysortedbythecolorname.Eachof
thosethreewouldpointbacktotheonedocument.Similarly,therewillbethree
entriesforTristanandHalli.
You can also set up an index even if the array contains elements that are
objects,aswiththefollowingexample:
{"name":"Skylar","clothes":[
{"type":"schooldress","color":"tan","size":5},
{"type":"schoolpants","color":"tan","size":6},
{"type":"churchshoes","color":"white","size":3}]
{"name":"Korver","clothes":[
{"type":"pajamas","color":"green","size":3},
{"type":"tennisshoes","color":"brown","size":2},
{"type":"wintercoat","color":"blue","size":3}]
The following example will set up an index just for one property on the
objectinthearray:
{"clothes.color":1}
Multiple-propertyindex
Many times, you will have queries that specify more than one property to
matchon.Thisisusedwhenyouwanttonarrowdownyoursearchevenfurther.
Whatifyouwantedtorunbookstorespecialstoencouragepeopletouserewards
points.
Forexample,whatifyouwerelookingforallpeoplelessthan20yearsold
thathadnorewardspointssoyoucouldgivethemsomepointsasafreebonus
offertotrythemout.Thefollowingexamplecreatesanindexonbothageand
rewardsPointssoyoucansearchforcustomersthatway:
{"age":1,"rewardsPoints":1}
This is called a compound index. Be sure to understand how this really
works.Theunderlyingindexhassortedentriesforageandthen,hassub-sorted
entriesbyrewardsPoints.Thismeansthatyoucannotusethisindexforaquery
forjustrewardspoints.Findingallpeoplewithover1000pointswillnotbeable
to use the index. You can, however, use this index to query just by age. This
meansthattheorderofpropertieslistedfortheindexisimportant.
Youcouldcreatetwoseparateindexestobeabletoquerybybothageand
rewards points separately as well as combined in either order. MongoDB will
usesomething called an index intersectionfor you if it can.For example, you
couldcreatethefollowingtwoindexes:
{"age":1}
{"rewardsPoints":1}
Withthis configuration, you can still query by the combinationof age and
rewardspoints.Butyoucanalsoqueryjustbyage,orjustrewardspoints.You
canalsoreversethecombinationandquerybyrewardspointsandthenage,in
thatorder.Theonlydownsideisthateachindexyousetupmeansmorestorage
overhead.
Indexsortorder
Uptothispoint,Ihavealwaysbeenusingthenumericvalueof1inallofthe
indexexamples.WhatthatisdoingisinstructingMongoDBtocreatetheindex
in ascending sort order. You can alternatively specify -1 and get a descending
sortorder.Forexample,youcouldcreatethefollowingindex:
{"age":-1}
Forsingle-propertyindexes,thisdoesnotmatter.Itonlymattersifyouwant
toreturnmultipledocumentsofmorethanonepropertythroughacorresponding
multi-propertyindexandyouwanttheresultreturnedinaspecificsortorder.For
example,queryingforallpeoplewiththelastnameof“Smith”andreturningthe
documents in descending order by age. Otherwise, you can certainly add a
commandtosorttheresultthatisreturned.
With the API usage, there is a sort() function that can follow the find()
functionthatcanprocessthesortorderforyou,butitwillbedoneoutsidethe
indexandbeslower.
Othertypesofindexes
Theindexexamplesusedsofarwouldbeusedforexactvaluematchesand
rangetypequeries.Justsoyouareaware,thereareafewothertypesofindexes
thatyoucaninvestigateandutilizethatmaysuityourneeds.Someoftheseother
indextypessupportedbyMongoDBaregeospatial,hashedandtextindexes.
Forexample,let’ssayyouwerekeepingadatabaseofrestaurantsalongwith
their menus, reviews, and location coordinates. With a geospatial index, you
could then perform queries to take a person’s current location and find all
restaurantswithinacertainradiusofthem.
Ifyouhavelargeamountsoftextinaproperty,youcanusewhatiscalleda
textindex.Forexample,ifyouwerestoringnewsstoriesandyouwantedtohave
anindexonthestorycontent,youcouldfindatextindexveryperformant.
Thereisanindexcalledahashindex.Itcanbeusedonpropertiesthatholda
single value (i.e. string, number) or that have an embedded document, but it
cannot be used with array properties. The hashed index is populated with the
hashedvalues.Ahashindexbyitselfdoesnotworkforrange-basedqueries,so
youmightwantanotherseparatesingle-propertyindexforthat.
Forembeddeddocuments,theindexgivesahashofthecompleteembedded
document. This then would potentially be a faster lookup as the search is just
comparinga hashvalue to findthe document.Ahash indexmight make more
sense when combined with sharding. This will be covered in an upcoming
chapter.
Indexcreationoptions
Oneoftheoptionsyoucanusewiththecreationofanindexistospecifythat
the values for a given property are unique. For example, in the bookstore
example,eachcustomerhasanemailproperty.Youcouldmaketheindexrequire
that all emails must be unique across all documents. Here is an example that
createsanindexwiththeuniqueoption:
{"email":1},{unique:true}
You should create an index like this before any data is populated. Index
creationwouldfailifmultipledocumentsexistthathavethesamevaluefortheir
emailproperty.Onceanindexiscreatedwiththeuniqueoption,anydocument
creation will fail unless it has a unique value on the indexed field with this
optionset.
Thesparseoption can be used to create an index that only has entries for
documentsthat contain that property. In the following example, if a document
didnothavetheemailproperty,thenitwouldnotbeincludedintheindex.Any
subsequent query for values to match on using the email field would ignore
thosedocuments,butthatisprobablywhatyouwanted.Ifanindexexistsfora
property, then MongoDB will use it, so that is why the rest of the documents
wouldnotbesearchedthataremissingthatproperty.
{"email":1},{sparse:true}
Indexcreationoptions
That about wraps up the topic of indexes. As I stated at the start of this
chapter,therearenoperfectsolutionsintheworldofdatabases.Thisistruewith
indexing. What you need to do is to completely understand what your query
needsarefirstinordertounderstandhowbesttocreateyourindexes.
For example, you might even consider not having any index under certain
circumstances. If you had a database with 95% of the operations being writes
and5%werereads,youmightnotwanttocreateanindex.Thisisbecausean
indexwillslowdownyourwriteoperations.Thereare,however,configuration
settingsyoucanstillmaketospeedupwrites.
Ifyouhavetheoppositesituation,andreadperformanceneedstobefast,and
writesareasmallpercentageoftheload,thendefinitelycreateindexes.Youcan
always measure your performance before and after to make sure that your
indexingdecisionsarevalid.
Note: Once your database is up and running and you are running code
queries against it, you can get a diagnostic report on an individual database.
Thiswilltellyouhowwellyourindexisperforming.Thisisdonebyeitherusing
theexplainoptionortheexplain() method, dependingon the APIcall you are
making.
5.2AvailabilitythroughReplication
Asinglepointoffailureisnevergood,nomatterwhatserviceyouareusing.
Forexample,inthedaysofthetelegraph,theremighthaveonlybeenasingle
telegraphlineconnectingtwocities.Thatconfigurationcreatesaservicethathas
a single point of failure. Cut the single line and communication is severed.
Havingtwotelegraphlineswouldgiveyouredundancy.Itwouldalsogiveyou
greaterthroughputifyouputbothintousageatthesametime.
Thesameredundancyisnecessarywithdatastoredonharddrives.Perhaps
allyourfamilyphotosareonasingleharddrive.Whatifthatharddrivefails?
Doingbackupstomakecopiesisnecessary.Anyapplicationdatamustneverbe
at risk of being lost or being unreachable. Therefore, some form of data
redundancyisnecessarywithMongoDB.
MongoDBreplicaset
If you have a single machine for your MongoDB database, you could
configure a MongoDB single-node. Then when that goes down, you cannot
accessyourdatabaseuntilitcomesupagain.Thiswouldbefineforoccasional
usagescenariosorfordevelopmentexperimentation.
MongoDBhastheabilitytoconfigurewhatiscalledareplicaset.Thisgives
youmultiple parallel,redundant copies ofall data. Thisensures that yourdata
willbe safe and available.This is what youalways get by defaultthrough the
AtlasportalifyouutilizethatPaaS.
Inareplicaset,youhavemultipleduplicateddatabases.Onlyonemachineis
designatedastheprimaryatanygiventime.Iftheprimarydatabaseservergoes
offline, a secondary server will take over. The following diagram gives you a
generalideaofwhatthislookslike:
Figure18-MongoDBreplicaset
Theconfigurationwilllookslightlydifferentbasedonwhichplanyouselect
from Atlas. The basic idea is that the primary database server receives and
fulfillsallreadandwriterequests.Allthewhile,thesecondarydatabaseservers
are kept up to date with all changes. Each database server can be on its own
dedicatedAWSEC2virtualmachineindifferentavailabilityzones.
Each server is constantly being checked with a heartbeat signal. If the
primary database goes down, the two secondary servers and the arbiter would
detectthat,andoneofthesecondaryserverswouldbeswitchedovertobecome
theprimarydatabaseserver.Thearbiterisreallyjusttheretobreakanytievotes
ifneeded.
Areplicasetallowsforfasterreadingofdatabecausemultiplecopiesexist
anddata can be fetched in parallel from each replicacopy, if that is what you
want.Youmust designatereads tobe fulfilledby the secondaryservers if you
determinethatisjustified.Justbeawarethatyoucouldgetstaledatathathasnot
yetbeenupdatedbyareplicationprocess.Thisisgreat,becauseyoukeepadding
secondarymachinesandcanachievebetterreadperformancenow.
Note: With a PaaS solution, you generally do not have control over the
replica set configuration unless you work with the provider to get something
customized. There are pre-determined configurations you select when you
purchaseaplan.NothingispreventingyoufromimplementinganIaaSsolution
andsetting up your own virtual machines and replica set configurationif that
wouldworkbetterforyou.
Secondaryconsistency
Thereisacomplicationtobeawareofinhavingreplicationsavailable.Any
writetotheprimarystoragecollectionmusteventuallymakeittoallthecopies.
Therefore,youmustmakeachoiceastohowthatreplicationisaccomplished.
MongoDBhasasettingcalled“writeconcern”thatallowsyoutospecifyif
youwantamajorityofreplicastoreportthatthewritehastakenplacebeforeit
isacknowledgedorfailed.Youdon’thavetorequirethis.Ifyoudon’t,thenall
writeseventuallymaketheirwayasynchronouslytoallreplicadatabaseservers.
5.3Sharding
The replication previously discussed stores the same data on multiple
machines to provide emergency backup to ensure availability. Sharding also
spreadsdataoutacrossmachines. Withsharding,agivendocument appearsin
only one replica set of a sharded cluster, but would be in the primary and
secondarymachinesofthatshard.
Thepurposeofshardingistoallowyoutogrowtheamountofdatayoucan
store and also increase the performance of operations. Both concepts of
replication and sharding can be applied at the same time in an architecture.
Sharding is just the increasing of the number of replica sets that you have as
individualunits.
Shardingspreadsthedata acrossmultiplereplica sets.The multiplereplica
setsinashardedclusteractasiftheywereonesinglecollection.Thesharding
technologyknowswheretogoforanygivenreadorupdatetomakeiteasyfor
youtouse.
Sharding helps when you have large datasets and are wanting to maintain
high throughput. For example, you might have a lot of data constantly being
accessed.ThiscanbecomeabottleneckwithasingleSSD.Ifyoudistributethe
load across multiple SSDs, then the CRUD operations would not conflict as
much.
MongoDBcanbesetuptotakecareofeverythingforyou.Youcanselecta
planfromAtlasthathasitallsetupforyou.
Atlashasvariousplansyoucanchoosefrom,dependingonhowmuchyou
are budgeting to spend. With the free-tier plan, there is a hard limit with one
singleSSDblockstorageforyourdatabase,soyoucanonlygouptoacertain
sizeandthenyoucan’tgrowbeyondthat.Atlascurrentlyonlyoffersreplicaset
and sharded plans (multiple replica sets), and you can’t just have a single
MongoDBserviceonitsown.
Clusterplansgouptoacertainamountofstorage.However,workingwith
mLaborAtlassupportpeople,youcankeepincreasingthehorizontalscalingof
theshardingby addingmore storage.Additionalreplica setscan, intheory,be
addedtoaccommodateyourlargestdatastorageneeds.
Reasonsforsharding
Theconceptofdatasharding(alsocalledpartitioning)wasinventedtohelp
approachtheidealofbeingabletostorean“infinite”amountofdataandretrieve
any part of it in a minimal amount of time. Let’s dig a little deeper into the
scenarios that will cause you to implement a strategy for sharding. Here are
somereasonstoimplementdatasharding:
Runningoutofroom:Withalimittostorageforasingledatabase
SSD,youmightsimplyoutgrowthatcapacity.
Machine performance: There are utilization limits for RAM,
CPUandSSDaccessthatmightbereached.
Note: With an Atlas PaaS databases, you have the ability to choose a
configurationwithshardingalreadyconfiguredforyou.InthecaseofmLaband
Atlas,you can picka preconfigured machinearchitectureand then setup how
yourshardingwillact.IfyouwanttogotheIaaSroute,thenyoumustconfigure
thisyourself.
Howshardingworks
Here is how sharding works. Imagine that you start out with a single
MongoDBdatabaseserverandonthatserver,youhaveasinglecollection.Each
document you create could have a property that has a random capital letter
chosenfromAthroughZ.Youmightalsosetupanindexontheletterproperty.
AJSONdocumentyoumightwanttoinsertcouldlookasfollows:
{
"letter":"G"
}
Atthispoint,nomatterwhattheletterpropertyvalueis,alldocumentswill
be created in the same database collection. The box below represents a single
replicaset(primaryandtwosecondarymachines).Thisexampleshowswhatthis
wouldlooklike:
Figure19-Collectioninasinglereplicaset
Then, at some point, you realize that you need to add a whole lot more
documentsand want to achieve ahigher level of throughput on yourread and
writeaccess.Theabovesingle-nodeconfigurationcanthenbemadeintowhatis
calledamulti-nodeshardedcluster.
MongoDB will start balancing documents between the available shards
(replicasets)intheclustertocreateamoreevenlydistributedstorage.Itactually
does this in chunks. With additions and deletions of documents happening,
MongoDB keeps it all balanced based on the sharding key. You can choose
eitherahashorarangestrategyforyousharding.
Yourdocumentsendupbeingdistributedoverthethreeshardsinthecluster.
See the following figure for a visualization of the distribution using a range
shardingstrategy:
Figure20-Collectiondistributedoverthreeshards
Asitturnsout,eachshardisareplicaset.Whenadatabaserequestcomes
intothecluster,MongoDBdoesalltheworktoroutetherequesttotheproper
replica set shard. Your code is shielded from the fact that this is going on. A
singlelogicalcollectiondoesalltheworkforyoutocoordinateacrosstheactual
shardsthathavetherealcollections.
Iwon’tgointothearchitecturaldiagramshowingthecomponentstosetthis
up, but you can look it up online if you really want to implement an IaaS
configurationonyourowninsteadofusingthePaaSsolution.Whenyouusethe
AtlasPaaSsolution,youwouldmostlikelyenlistasupportengineertohelpyou
ifyouwantedtocustomizeyourshardedcluster.
Note:Thereisawaytotakeshardsoutofyourcluster.Thereisamechanism
to let MongoDB know that this is your intention. Once you do so, MongoDB
beginsmigratingdataoffofthesoontobedecommissionedshard.Oncethatis
done,thatshardcanbefreedup.
Shardingkey
Yourshardmustbesetupwithwhatiscalledashardingkey.Ashardingkey
issimilartohowanindexissetup.Withanindex,youspecifyapropertythat
youwanttouseforaspeedylookup,usingsomedeterminedalgorithm,suchasa
rangeorahashsearch.
Lookatthepreviousfigure,andyouwillseethreeshards.Theshardingkey,
inthiscase,isthepropertythatMongoDBwillusetodeterminewhatshardeach
documentexistsin.Forthisexample,itwouldhavebeentheletterproperty.
Eachdocumentcanonlyexistinonesingleshard.Ashardkeyisthusused
asasortingproperty.IfIcreatedadocumentwiththeletterpropertysetto‘M’,
it could be stored in the middle shard because of the range strategy setting it
there.
There is a fair amount of thought needed to select the proper sharding
strategyandselectapropertytokeyoffof.Justrememberthatyoumustknow
whatyourqueriesaregoingtolooklike.Don’tforgetthatyoumightevenhave
queries that cross shards, like those using range criteria. Imagine if you want
documentsfromthepriorexamplethathadalettergreaterthanDandlessthan
L?Theshardservicewouldactuallyknowitneedstosendthequerytoboththe
firstandsecondshardsandthenyourcodewouldprocessalloftheresultsetfor
whatyouwant.
Indexesstillexistoneachshard.Thequerylookupwouldfirstgotoashard
and then the shard replica set would use any applicable indexes to find the
document(s). You can have multiple indexes, but only one sharding key. The
shardingkeymustbethesameasoneoftheindexes.Inourexample,therewas
anindexfortheletterproperty,andthatwasalsousedfortheshardingkey.
Ifyouhaddocumentsrepresentingcustomers,youcouldlookpeopleupby
theirlastname.Youcouldthenuseahashedshardingkey.Thatway,queriescan
narrow the location to one single shard and then quickly retrieve documents
fromthat shard using the index. A hash shard key is nice because it can most
likely give you a more uniform distribution of documents across shards for a
fairly even retrieval cost. This is great for locating documents with a specific
querythatcanzeroinonthedocument.
Rangequeriesmightnotbeaseffectivewithsharding.Ifyoudoknowyou
haveagooddistributionofrangevalues,thenperhapsarangestrategywouldbe
best. Range sharding is efficient if you have queries where reads target
documentswithinacontiguousrangeofvalues.
You also must consider what your queries will look like and what your
document composition will look like. For example, you will certainly have a
performanceproblemwithhashstrategyshardingifyoutryanddoarangetype
ofquerythatcausesalltheshardstobesearched.
Ifyourquerydoesnotactuallyutilizetheshardkeyproperty,thentheservice
hasnochoicebuttosendthequerytoallshardsintheclusterandthencollectall
oftheresults.Butatleast,youshouldhaveconsideredwhattheindexshouldbe
formakingthateffective.
Likeanindex,ashardkeycanconsistofmultiplepropertynames.Youcould
thus use a compound key such as last name, first name, and city. Shard keys
cannotbecreatedforapropertythatisanarray.
Youmight have been thinking that if you kept adding documents with the
sameorsimilarkeyvaluethattheycouldallgotothesameshardandthenthe
diskforthatwouldrunoutofmemory.Thisisactuallynotthecase.Thereisa
process going on in the background, regardless of the sharding type (range or
hash) that moves data around between shards in chunks. This is called the
Balancer. You don’t actually set the shard boundaries yourself. MongoDB
figuresthatoutforyou.Figure20wasjustafictitiousillustrationofapossible
balancingthatcouldhappen.
Aswasmentioned,youareultimatelylimitedbythediskspaceavailableand
thatis whyyou wouldkeep adding shardsas youreach thelimit of document
storage. As a new shard is added, the Balancer does the work to spread
documentsevenlyoutacrossallavailableshards.
Note:Letmemakesureyouhavealltheterminologydown.TheAtlasportal
letsyoucreateacluster.Thisisthesetofmachinesthatholdyourdatabases.In
theirterminology,aclustercanbeeitherareplicaset(primaryandsecondary
machines), or a sharded cluster (multiple replica sets that are each called a
shard)
Chapter6:NewsWatcherApp
Development
This chapter takes some of the concepts that you have learned and applies
themtoaprojectusingtheAtlasPaaSoffering,hostedinAWS,tocreateadata
layer.Youwill learnhow toget thedata layerup andrunning and learnsome
bestpracticesalongtheway.
In this chapter, you will go to the Atlas portal and create the MongoDB
database and collection resources for the NewsWatcher sample application. To
getstartedyoumustfirsthaveanactiveAtlasaccount.
Note: MongoDB is an open-source project and you could download it for
freeandrunitonanymachineyoulike.Thisisnottheapproachtakeninthis
book. You can certainly investigate that option if it better meets your needs.
ThereareotherMongoDBPaaShostingoptionsouttherebesidesAtlas,sodo
yourresearch.
Youwillbesettingupthefollowingresources:
Figure21-NewsWatcherMongoDBresources
Theonlydocumentyouwilladdtothecollectionrightnowistheoneyou
shouldaddmanually.ItisrequiredforthefunctionalityoftheNewsWatcherapp.
Youshouldalsomanuallyaddafewotherdocumentsjustfortestingpurposesto
tryoutafewqueries.Later,youwillseehowdocumentswillbeaddedthrough
JavaScriptcodeinyourNode.jsprocess.
6.1CreatetheDatabaseandCollection
Thefirsttaskwillbetocreatethedatabase.Forthesampleapplication,you
canselecttheoptionthatwillgiveyoufreehosting.Thiswillbefineforyour
development and testing purposes until such time that you need to scale for
greaterstorageandperformance.
You can also study the other configuration offerings available through
MongoDB Inc. You can even try them out for a day or two, as you are only
charged for the time you have them available, and you can easily delete them
whenyounolongerwantthecharge.
Itiscertainlyworththecosttotryoutsomeoftheotherconfigurationsthat
allowforothercapabilitiessuchassharding.Youmightwanttotakesometime
to look through the plans and pricing pages on the MongoDB Atlas site to
familiarizeyourselfwithwhatispossible.Forexample,theamountofRAMand
storageand IOPs changes per the differentplans. Thus, it is veryimportant to
understandyourusageneedstobeabletoselectaplan.
When you sign up, you can select the hosting provider such as AWS, the
hosting location, and the specifics about the configuration that determine the
charge. To create a database cluster for the NewsWatcher app with the Atlas
portal for free hosting, you can go to the MongoDB Inc. website
(https://www.mongodb.com/). It is fairly self-explanatory from there. First,
createaprojectinyouraccount.ThenlookfortheBuildaNewClusterbutton.
Atsomepoint,youwillneedtoprovideausernameandpassword.
Aftercreatingyour databasecluster everythingwill bedeployed andready
foryourusage.Onceitisreadyyouneedtosetupaccessonyourmachinefor
theCompassdesktoptoolortheMongoshell.
1. ClicktheConnectbutton.
2. ClickADDCURRENTIPADDRESSandgiveitaname.Ifyour
IPaddresswillbeconstantlychanging,selecttoallowallIPaddresses.
You will still be secure, as access must be authenticated and
authorized.
3. DownloadtheMongoDBCompasstoolfoundintheAtlasportalat
thebottomofthepage,byclickingDownloadCompass.Thiswillbe
necessarylaterforyourinteractionswiththedatabaseandcollection.
ThefigurebelowshowsyouwhattheAtlasportallookslikeatthetimeyou
useittocreateadatabasecluster.
Figure22-CreateClusterpage,MongoDBInc.portal
Inafewmoments,youwillbeallsetupandwillbereadytostartusingyour
freeMongoDBdatabasefromMongoDBInc.,hostedonAWSEC2machines.If
youlookintheMongoDBAtlasportal,youwillseeyourclusternowshowsup.
SetupaconnectionwithCompass
WiththeCompasstoolinstalled,youwillneedtomakeaninitialconnection
toyourMongoDBhosteddatabase.Lookbackattheinformationyousawinthe
Atlas portal when you clicked CONNECT for your cluster. In there you will
find the hostname of your cluster. You can configure Compass to be able to
makeaconnectionasfollows:
1. In the Atlas portal, click the CONNECT button again and then
clicktheareathatsaysConnectwithMongoDBCompass.
2. ClicktheCopybutton.
3. OpentheCompasstool.
4. Clicktocreateanewconnection.Everythingshouldauto-populate,
andyoushouldbegoodtogobyclickingCONNECT.Ifnot,follow
thenextinstructions.
5. PasteintheHostname.
6. Theportshouldalreadybesetto27017.
7. SetAuthenticationtoUsername/Password.
8. Entertheadminusernameandpassword.
9. LeaveSSLtoUseSystemCA/AtlasDeployment.
10. LeaveSSHTunneltoOff.
11. EntersomethingtorememberthisconnectionbyfortheFavorite
Name.
12. ClickCONNECT.
Figure23-CompassconnectionUI
AddadatabaseandcollectionthroughCompass
You need to create a database with a collection inside of your cluster as
follows:
1. OpentheMongoDBCompassappandconnect.
2. ClickCREATEDATABASE(hoveroverthe“+”plussignatthe
bottom of the left pane) and type in the Database Name. I entered
“newswatcherdb”. Enter a name for Collection Name. I entered
“newswatcher”.
3. ClickCREATEDATABASEintheform.
Note:IfCompasswillnotallowyoutoperformedits,itisprobablybecause
youareconnectedtoasecondary,andnottheprimarymachineinthecluster.
You can now create the one required document that must be manually
created.Youwillseelaterhowthisdocumentfitsintoyourdatamodel.Tocreate
thedocument,dothefollowing:
1. Clickonthenewswatcherdbdatabase.
2. Onthenewswatchercollection,clickINSERTDOCUMENT.
3. Typeinthedocumentcontentasshownbelow.Makesuretoselect
adatatypeofArrayforthenewsStoryproperty.ClickINSERT:
{
"_id":"MASTER_STORIES_DO_NOT_DELETE",
"newsStories":[],
"homeNewsStories":[]
}
Figure24-Createadocument
Toaddanewproperty/field,youcanhoveroveranumberontheleftandit
willchangetoaplussignandthenyoucanaddone.Youalsoneedtochangethe
typewiththedropdownyouwillfindthere.AfterclickingINSERT,youwill
seethedocumentonthecollectionpage.Isn’tPaaSwonderful?Thereisnosetup
ormaintenanceorworryingaboutifyouhavethelatestversionofMongoDB.
Installofthemongoshell
The Compass tool does not now allow for easy creation or import of
documents,becauseyou havetocreate themaproperty atatime. Toimporta
largerdocumenteasily,youneedtoinstallthemongoshell.IntheAtlasportal
UI,youwillfindalinktodownloadthemongoshell.Youwillwanttoselecta
custominstallwiththeselectionslookingasfollows(seefigure25).Otherwise,
youwillgetthedatabaseinstalledlocallyalongwiththemongoshell.Youdon’t
actually need the mongo shell to go forward developing the NewsWatcher
application,sodon’tinstallitifyouarehesitantaboutit.
OntheUIpageofAtlasthatshowstheconnectioninformation,youwillfind
astringtocopythatgivesyoutheinformationnecessarytomakeaconnection.
Givethefullpathtothemongoexecutableandaddtheconnectionstringtothat.
Itlooksasfollowstorunthemongoshell:
"C:\Program Files\MongoDB\Server\3.4\bin\mongo.exe" "mongodb://cluster0-shard-52-78-
k6yhs.mongodb.net:27017, cluster0-shard-52-78-k6yhs.mongodb.net:27017, cluster0-shard-52-78-
k6yhs.mongodb.net:27017/test?replicaSet=Cluster0-shard-0" --authenticationDatabase admin --ssl --
usernamemax--passwordblah
Figure25–Installthemongoshell
6.2DataModelDocumentDesign
It is time to diagram out the structure and relationships of the document
typesthatyouwillneedfortheNewsWatcherapplication.Thisisdefinitelyan
iterative process where refinements are made over and over until it is correct.
Evenafter you have implemented a data model, you may find that it does not
give you the performance you expected, and you might end up altering the
design.
Think again about what the requirements are for the NewsWatcher
application and you can understand what is needed. NewsWatcher will have
users that log in. Thus, you have identified that there is a need for a user
document.
There is also a single document that holds the master list of news stories.
Therewillbesomecodethatisruneveryfewhourstocollectnewsstoriesand
storetheminthatdocument.
Athirddocumenttypeisforthenewsstoriesthatusersshareandcomment
on. There would be multiple User and SharedStory documents, but only one
MajorStoriesdocument.Thismodeliscompletelydenormalized,sothereareno
keystolinkanydocumentstogetherandtherewillnotbeaneedforanytypeof
joinoperations.Thefollowingdiagramshowssomeoftheneededdocuments:
Figure26-NewsWatcherdocuments
Let’s look at what the User document contains. In there, you will want to
includeanemailaddressforeachuser.Thiswilluniquelyidentifyyourusersand
allowsthemtosignin.Usersmustalsoenterapassword.Youcansafelystorea
hashedvalueofthepassword(youshouldneverstoreapasswordinplaintext).
Thenyoucanletuserspickadisplaynamethatotheruserswillseewhenauser
commentsonasharedstory.Youshouldneverrevealtheiremailtoanyoneelse.
Next, there should be certain global values that can be used for user
preference settings. You can put that in a sub-hierarchy called settings. For
example,you might want to give users the option of not using any cell phone
dataandrestricttheapptousingWi-Fionly.
Youcanassumethattherewillbesomeusersthatwouldlikeanalertfeature
for when news stories come in to be immediately notified. Youcould create a
Booleanvalueforthat.
ThecompellingfeatureofNewsWatcheristheabilitytohavetheappscan
forthenewsausercaresabout.NewsWatcherusersarenotthetypethatwants
to go to some general overall curated news page, but are interested in
customizing their own specific filtered view of their news. This is done by
filteringnewsstorieswithkeywords.
Userscan setup asmany filtersas theylike, soyou canconclude thatthe
designrequiresanarrayoffilters.Eachfilterwillneedtocontainatitleforthe
filter,keywords,timeofthelastnewsscanandalistofstoriesandtheirtimeof
capture.Thelistofstoriesforafilterispopulatedbyscanningthemasterstory
documenttoseeifthereareanymatcheswiththekeywords.
NewsWatcherhastheabilitytosaveoffinterestingstoriespereachuserso
thatthey appearseparately.This is what the savedStories property is used for.
Wewon’tactuallyimplementthatatthispoint.
The other properties shown in the user document are for other features as
outlinedintherequirements.Thoseotherfeatureswon’tbeimplementedthough.
ThiswillgiveyouagoodstartatanMVP(MinimumViableProduct)togo
outwith.Ifyoudoabitofadvancedthinking,youcanmodelallofthisinyour
diagramandjustnotimplementeverythingyet.Youcanfeelconfidentthatyour
datamodelcanaccommodateyourfutureneeds.
Enteringsometestdata
Atthispoint,youcanopenthepageforthenewswatchercollectionandadd
adocument to the collectionfor testing thingsout. Youcan of course onlydo
this if you had done the work to install the Mongo shell. You can insert
documentswithCompass,itwilljusttakeyoulonger.
1. Startthemongoshellasshowninsection6.1.
2. Attheshellprompt,youcantypeusenewswatcherdbtodirect
commandstothatdatabase.
3. Runthefollowingcommandtoinsertadocument:
db.newswatcher.insertOne(
{
"type":"USER_TYPE",
"displayName":"Bushman",
"email":"nb@hotmail.com",
"passwordHash":"XXXX",
"date":1449027434557,
"settings":{
"requireWIFI":true,
"enableAlerts":false
},
"savedStories":[],
"filters":[
{
"name":"TechnologyCompanies",
"keyWords":[
"Apple",
"Microsoft",
"IBM",
"Amazon",
"Google",
"Intel"
],
"enableAlert":false,
"alertFrequency":0,
"enableAutoDelete":false,
"deleteTime":0,
"timeOfLastScan":0,
"newsStories":[]
}
]
})
4. Run“db.newswatcher.find()attheprompttoseeanydocuments
inthecollection.
5. Run“exit”toquitthemongoshell.
You will see the document added. It has an automatically assigned _id
created. You can launch the Compass application and view your created
documentsanddeleteoreditthemasnecessary.
For now, you can go ahead and experiment by creating a few more User
documentsinthissamecollection.Later,documentswillonlybeaddedthrough
code. At this point, all you are interested in, is being able to test out some
queriesbeforedevelopingthenextlayeroftheapplication.Youcangetafeelfor
howtheportalUIisusedandlearnabouthowqueriesareconstructedbeforeyou
putthoseintocode.
IfyoureadthroughtheMongoDBdocumentationyouwillfindthatthereare
ways to do bulk importing or exporting of documents. For example, there are
toolslike mongoimport andmongoexport that youcan run from thecommand
line.Forexample,anexportmightlookasfollows:
mongoexport -d test -c records -q '{ date: { $lte: new ISODate("2017-09-01") } }' --out
exportdir/myRecords.json
6.3TryingOutSomeQueries
Youmighthaveenteredafewdocumentsbyhandinacollection.Youcan
nowtryoutsomequeriesagainstthatdatathroughtheCompassapplicationor
theMongoshell.Atthispoint,youjustwanttogetafeelforwhatthetoollooks
likeandtobereadytolearnabouthowqueriesareconstructedbeforeyouput
thoseintotheservicelayercode.
You will use the sameUI shown in earlier chapters to run queries against
yourMongoDBcollection.Thatiswhereyouutilizethecriteriasyntaxtoquery
andspecifywhatyoudesiretoseeintheoutput.
Trysomequerieslikethefollowing:
{"type":"USER_TYPE"}
{"type":"USER_TYPE","email":"nb@hotmail.com"}
Now set the PROJECT in the options area of Compass to be
{"displayName":1}andtrythequeryagain.
Ifyourquerysyntaxisincorrect,youwillbenotifiedoftheerror.However,
ifyoumistypethenameofapropertyyouwanttoprojectorqueryfor,youwill
not get an error but will get an empty result instead. For example, try the
projectioncriteriapropertynameas{"blah":1}.Ifyoudothis,youwillnotget
anerrorbutwillgetanemptyresultset.
KeepinmindthatMongoDBisaschema-lessdatabaseanditassumesthat
the“blah”propertycouldbethereinthefuture,butitjustisnotthererightnow.
Propertiescancomeandgoinaschema-lessdocument-baseddatabase.
6.4IndexingPolicy
Youcould writecodethat usestheMongoDBAPIthat runsto createyour
neededindexes.Myapproachistonotputthingsinthecodethatareone-time
configurations. I will instead prefer to use the Compass application to create
indexes. You could also use the mongo shell to run a command to create the
index.
For the NewsWatcher application, you can imagine that you would have a
queryintheservicelayerthatwilllookupauserbytheiremail.Youwouldwant
toaddaspecificindexforthatbydoingthefollowing:
1. OpentheCompassapplication,clickthenewswatchedbdatabase.
2. Clickthenewswatchercollection.
3. ClicktheINDEXEStabthenclickCREATEINDEX.
4. Givetheindexaname.
5. Select“email”asthepropertyand“1(asc)”.
6. SelectthecheckboxforCreateuniqueindex.
7. Select the check box for Partial Filter Expression and enter
“{email:{$exists:true}}”
8. ClickCreate.
You check Create unique index as you don’t want email addresses
duplicated across users. This is a way to uniquely identify an account for a
person. You have also set up what can be called a “sparse index” using the
PartialFilterExpression.Thismeansthatdocumentsthatdon’thavethe“email”
property will not be used in the index. This will give you lower storage
requirementsandofferbetterperformancewiththeindexmaintenance.
Figure27-Creatinganewindex
6.5MovingOn
Thiscompletestheworktogetthedatalayerupandrunning.Youcansee
thatitwasallaboutsettingupyourconfigurationthroughtheAtlasmanagement
portalandtheCompassapp.Youdidnotneedtowriteanycodeyet.Thismeans
that you are postponing the writing of any Node.js JavaScript server-side
functionalityuntilyouworkontheservicelayer.
Testing
For the NewsWatcher application, the service layer is actually the proving
ground for the data layer. The service layer will connect directly to the
MongoDB collection and perform CRUD operations. There will be functional
testsputin placeto provethat thedata modelworks. Youwillalsobe ableto
takecareofthenuancesthatgoalongwiththedatalayer,suchasperformance
tuningandconcurrencyissues.
In reality, the best way to develop software is to work on it in terms of
verticalslicesoffunctionality.Thismeansthatforanyfeaturesyouhavethought
up,youwouldimplementitinallthreearchitecturallayersatonce.
Chapter7:DevOpsforMongoDB
In this chapter, I will go over some of the operational responsibilities for
managing a MongoDB database. For example, with NewsWatcher, you know
that the data layer stores user accounts and news stories. You can think about
whatconcernsyouwouldhavewiththat.DailyDevOpsworkwill involvethe
monitoringofthedatabase.YoucantakealookatwhattheAtlasmanagement
portalwillletyoumonitor.
Youwillwanttomakesurethatdataaccessissecureandperformant.You
can set up replication, sharding, and indexes. Once those are set, you should
leavethemaloneuntilsomechangecomesalongthatcausesyoutotweakthem
foraspecificreason.
7.1 Monitoring through the Atlas
ManagementPortal
Oneviewyoucanlookatistheviewofmachinesinyourdatabasereplica
set.Hereisascreenshotthatshowsallthemachinesinareplicaset(primaryand
twosecondaries):
Figure28-Atlasmanagementportalserverview
Note:ThefreetierofferingislimitedinwhatitoffersintheAtlasportal.For
example,thereisaDataExplorerwiththepaidtier.TheMetricspageforapaid
tierhasalotmoreinformationavailablelike-ShardedClusterMetrics,Replica
SetMetrics(moremetrics),Real-TimeTab,StatusTab,HardwareTab,DBStats
TabandChartControls.
Telemetrycharts
You can drill down further into the performance metrics of each of these
machines by clicking on them. There are charts provided in the Atlas
management portal to show you the server utilization numbers. The real-time
telemetryvalueswillletyouknowthingslikehowmuchstorageyouhaveused
up.
Here is an image showing the Atlas management portal monitoring page
withthetelemetrythatisshownbydefault.Inareplicaset,youwillhavemore
than one machine, so you have to pick the primary or one of the secondary
machinesifyouwanttoseetheirtelemetryseparately.
Figure29-Atlasmanagementportalmetricspage
Telemetryalerting
You probably want to be aware of how much storage is left for your
database. You could set up an alert to notify you when you are reaching this
limit.Youmightalsobeconcernedaboutmachineperformanceandsetupsome
alerts around specific performance measurements. If you see your machine
performancedegrading,thenyoucanshifttoamorecapableconfigurationwith
thePaaSofferingsofMongoDBInc.
IfyougointoyourAtlasportal,youcanselectAlertsfromthemenuonthe
left.Thenyoucanselecttheappropriatetabfromthere.Forexample,thereisa
tabtoviewandacknowledgealertsthathavebeentriggered.
ClickontheAlertSettings tabto seewhat alertsyou haveby default and
add any you need. In the figure below, you can see that there is an alert that
triggerswhenyouhavereached90%ofyourstoragecapacity.
Figure30-AtlasAlertConfigurationpage
Youcanselectfromawidechoiceofpossibilitiesforhowyougetnotifiedof
an alert being triggered. Click on the Add button to see these. The selections
include Atlas User, Email, SMS, HipChat, Slack, Flowdock, PagerDuty, and
Datadog.
7.2TheBlameGame
Onceyourdataissecureandhasbeentunedforthebestperformance,you
really don’t need much in the way of day-to-day care anymore. Believe me
though when I say this your potential troubles are not over by any means.
Frommyexperience,youwillbespendingyourtimecaringfortheintegrityof
the data as much as anything else. This is especially true if there are a lot of
othersystemsintegratingwithyoursthattouchthedataatsomepoint.
Unfortunately, every time someone sees a data corruption problem, they
come to blame whoever is in charge of the DBMS. You will hopefully have
confidencethatmostofthetimetheaccusationsareunwarranted,andyouwill
be able to track the problem down to some supporting system. For example,
some external system that is feeding you data may suddenly have missing,
intentionally altered, or corrupt data. It is a good idea to put data validation
measuresintoplaceatallthepointsofintegration.
Youwouldalsobewisetoputsomehandyscriptsinplacetoallowyouto
diagnoseissuesandfixthem.Forexample,youmightneedtorecoverdatafrom
abackupsnapshotorreimportdatainbulkfromadependentsystem.
7.3BackupandRecovery
Thegoodnewsisthatyoudon’thavetoworryaboutdisasterrecovery.The
bad news is that you have to worry about disaster recovery. It all depends on
whatyourdefinitionisofa“disaster”.
Withareplicasetinplace,MongoDBstoresmultiplecopiesofyourdatathat
arealwaysinsyncwiththeprimaryserver.Thismeansthat,withinaregion,you
haveredundancyincaseofnetworkordrivefailures.Thisisthecasewiththe
differentAWSavailabilityzonesthateachEC2serverisinforyourreplicaset.
You have this set up for you through the PaaS plan you select and don’t
necessarilyhavetodealwithitdirectlyyourself.
This replication means that your data is safe from drive failures, machine
reboots,poweroutages,networkoutages,andsuch.Ifthedrivethattheprimary
copy of your MongoDB database is on goes bad, you are covered. MongoDB
and AWS will take care of rotating this drive out and moving you to a new
primarydriveandaddinganewreplacementbackup.
Youstillneedbackups
Datareplicationis notthesame asperforming adata backup.Just because
youhavereplicationdoesnotmeanyouareprotectedfromsomehowlosingor
mangling your own data by mistake. It is a good idea to institute a backup
processtoperiodicallystoreasnapshotofallyourdata.Thatwayyouareableto
recoverfrominadvertentcorruptionorlossofyourdata.
Backupsare useful in many scenarios. For example, you might have some
bugthatwasintroducedinyourcodethatcausesallyourdatatogetcorrupted.
You then need to roll back to the database copy you had before the data was
corrupted.
Youcould,forexample,writesomecodetocopydatafromoneMongoDB
regionintoanotherregionandkeepthatasabackup.
You could also save a collection as a file for safe-keeping on some local
machineyouhave,orplaceitinEBSorS3storageinacompressedform.Then
youcandoarestoreprogrammaticallyoruseatooltoimporteverythingfrom
yoursnapshot.YoucanexportaJSONfileandstoreityourselfifyoudon’twant
tospend the moneyon database collectionsor otheronline storage beingused
forbackups.
TheAtlasmanagementportalhasaBackupselectionontheleft-handside
thatletsyoucreateanimmediatebackup,ortoscheduleatimeeachdayforone
to automatically happen. There is a pre-determined retention policy for each
specifictime-relatedsnapshot. The coreMongoDB projecthas backup utilities
youcouldalsousetoperformdatabackups.
PARTII:TheServiceLayer(Node.js)
Parttwoofthisbookwillteachyouwhataservicelayeris.Youwillcreate
anHTTP/RestAPIthatinterfacestothedatalayer.Thiswillsetthingsuptobe
preparedforthedevelopmentof thepresentationlayerapplicationthattalksto
the service layer. Node.js/Express.js and JavaScript are the technologies of
choiceforthisservicelayer.
Therearemanydecisionsthatgointocreatingaservicelayer.Thefirstthing
to design, is what type of interface is needed over the data. This involves
separatingoutthedifferenttypesofdatathatyourRESTinterfacewillexpose.
ThisrequiresyoutothinkabouttheJSONpayloadsthatgettransferredbackand
forthforeachrequest.
You will learn how to use a Node module to call into the MongoDB data
layer developed in the first part of this book. At the end of this part, the
applicationwillbefullyfunctionalandreadytointegratewiththepresentation
layer.
Thisbook will be using Amazon Web Services (AWS) to host the Node.js
application.
Theextremelyimportanttopicoftestingwillbecovered,andyouwilllearn
how to use the Mocha test framework to run your tests. You will learn about
functionalaswellasperformanceloadtesting.
Inordertogivefullcoveragetothetopic,Iwillalsodiscusswhatitmeansto
setupallaspectsoftheday-to-dayoperationsofNode.jsforactualdatacenter
operationsmanagement.YouwilllearnhowtomanageaPaaSenvironmentand
howtododebuggingofissues.Securitywillbeanimportanttopicthatisalso
covered.
Note:ManypeoplerefertoNode.jssimplyasNode,andIoftendothesame.
Chapter8:Fundamentals
This chapter presents the fundamental concepts of the middle-tier of the
three-tierapplicationarchitecturethatisbeingoutlinedforyouinthisbook.You
willlearnwhatthemiddle-tieristypicallycomposedof.Icanthengetintothe
specificsandshowyouhowNode.jscanserveasamiddle-tierservicelayer.
8.1DefinitionoftheServiceLayer
The service layer provides the core capabilities of a three-tier architecture.
Thewholeideaofaservicelayeristobuildanabstractionlayeroverbusiness
logicanddataaccess.
Ifyoureallysimplifydowntheconceptsofathree-tierarchitecture,youcan
saythisaboutthelowerandupperlayers-thelowerdatalayerjuststoresdata
andtheupperpresentationlayerjustdisplaystheUserInterface.Thatleavesthe
middle-tierservicelayertodoalltherestofthework.Inmostapplications,you
willcertainly find morecode in the servicelayer than in theother layers. The
followingdiagramshowsthissimpleview:
Figure31-Simplifiedthree-tierarchitecturediagram
Itwouldnotbereasonabletohavethepresentationlayerhandlethebusiness
workflowlogic.Youalsodonotwanttoexposebusinessworkflowsinthedata
layer.Thedatalayershouldbekeptassimpleaspossibleandshouldonlyhandle
theCRUDoperationsandperhapsmoredifficultdatatransactionlogic.Youwill
normallyfindthemorecomplexbusinessservicescodeintheserviceslayer.
Note: Some architectures split the middle-tier out into a services and a
businesslayer.Thesetwoconceptsarecombinedinthisbook.Youwillfindthat
alltheneededfunctionalityofthislayercanbeaccomplishedwithinthesingle
technologyframeworkofNode.js,alongwiththeuseofnumerousnpmpackages.
Acontractofinteraction
A service layer can actually be created with no particular UI in mind. For
example,therearemajorcompaniesthatexpose theirAPIssothatanyone can
interface with their backend and write their own UI. Companies like eBay,
Twitter, and Facebook have been successful at this. For example, you can
interactwiththeeBayAPItobidonitems.
InthecaseoftheNewsWatchersampleapplication,thereisjustonesingle
UIthatIwrote,butanyoneelsecouldwriteadifferentUIontopofmyexposed
RestAPI.
Regardless of whether you are tied to one single front-end UI, or if your
servicelayerisopentoallowmanyapplicationstoconnect,youneedtothinkin
terms of a strict contract of interaction. This means you must define the
connectionroutesupfrontandtheJSONmessagesthatarerequired.
I will not be using any specific connection standard like you see with
standardssuch as SOAP.You canexplore things like Swagger on yourown if
youareinterestedinmakingyourAPIgenerallyavailabletopeopleinaneasyto
consumeandformalway.YoucanalsoexploreanAWSofferingknownasthe
APIGateway.ThiscanbeusedtosurfaceaformalAPIcontractthatsitsinfront
ofyourNode.jsservicelayer.
Anabstractionlayer
Aservicelayerisbuiltinawaythatabstractsawaythecomplexitiesthatgo
oninthebackend.Thiswillshieldthepresentationlayerclient-sidecodefrom
any tight coupling. The client is also protected from any backend rework. In
many cases, the client doesn’t even need to make any changes when backend
codeisrewritten.
Itmaybe,thatasinglecalltotheservicelayerresultsinaseriesofbackend
callsthatareeachprocessedinwhatcanbecalledaworkflow.Thiscoordination
fallssquarelyintheservicelayerinordertohidethecomplexity.Themultiple
backendservicescompriseyouroverallarchitectureandcanbeunifiedthrougha
singleNode.jsentrypoint.
For example, you might have one service that does all of the storage and
retrievalofall useraccount information.Another servicemight containbilling
account information, and yet another might deal with order information. You
shouldneverbuildonesinglemonolithicservicethatdoeseverything.Takethe
timetosplitupyourplatformsintodiscreetservices.Eachofthesewillservea
role, be self-contained, and operate independently. Research what is called a
microservicesarchitecture.
Thisdoesnotmeanthatexternalclientsneedtoauthenticateandconnectto
each of the microservices individually. In other words, you might want your
presentation layer to only have one single service layer entry point that
coordinates calls to other independent microservices that each exists as
autonomousservices. Youwillactually be gratefulyou have donethis, as you
canmakerathersignificantchangesinthelowerlayersandminimizethecode
thathastochangeintheclient.
Your backend might already consist of “legacy” systems that were not
written with Node. These backend systems might be written in different
languagesandberunningonyourownproprietary,on-premiseplatforms.Inthis
case,youcandecidetowriteagatewayservicelayerwithNodeandhavethat
code interface to your backend systems. Node is great for routing and
orchestration.Itcanworktoprocessrequestsasynchronouslywithahighdegree
ofconcurrency.AWSoffersanAPIGatewaythatyoucanalsoinvestigate.
AllorpartofyoursystemscanhavetheirownNode.jsinterfaces.Itisupto
you to decide what is worth your time and investment to do. The following
diagramillustratesthegatewaylayeringthatcouldexistinaserviceslayer:
Figure32-Servicelayergatewayconcept
If you decide you need to replace the billing system, the gateway service
providestheabstractionandprotectstheclientfromanychanges.
Servicelayerplanning
Knowing what operations and workflows are needed is the first step in
creatingaservicelayer.Usethefollowingquestionstohelpyoudeterminethe
designofyourservicelayer:
Whatoperationsandworkflowsareneeded?
Whatareyoursecurityandprivacyrequirements?
Howarepeopleauthenticatedandauthorized?
Is there a need for a pub/sub notification system to deliver push
notifications?
Isthereanytypeofdomain-specificconfigurationrequired?
Is programmatic resource management required? Scripting or
templates?
Arealldatainteractionsencrypted?
Aremultiplesystemsgoingtocallthis?Wouldamessagecontract
schemabeappropriate?
What validation can be made at the interface to not let anything
invalidin?
Whatmeaningfulerrorsneedtobereturned?
Doyouneeduserrolesandaccesscontrol?
Doyouneedcachingontopofyourdatabaselayer?
Is there any periodic processing of the data? Is it periodic in the
backgroundorreal-time?Inbatches?
Willyou need to queue up workand have workers process work
asynchronously?
WhataretheSLArequirementsforalloperations?
Whataretheaccessvolumeandtheratespertimeperiod?
Willtherebeburstsofactivityorisaccessevenlydistributed?
The answers to these questions should be carefully considered. Consult
experts along the way before you roll anything out into a full production
environment.
8.2IntroducingNode.js
The simplest way to describe Node.js, is to state that it is a runtime that
executesJavaScriptcodethatyouprovideit.YoumightthinkIjustdescribeda
browserenvironmentforyou.Afterall,thisiswhattheChromebrowsercando.
The Chrome browser has a JavaScript engine called V8 that is used for
executingclient-sideJavaScriptcode.
Node.js, however, is more typically running on the server-side. To
accomplish this, someone (Ryan Dahl) actually took the same V8 engine
mentioned,andmadeuseofitinaserver-sideprocessandthenaddedadditional
functionalitythatwouldmakesensetohaverunninginservercode.
You can think of Node as an abstraction layer on top of your operating
systemsoastomakeyourcoderunasplatformindependentcodeandgetmany
of the capabilities that an operating system provides (file system, processes,
networketc.).Node.jsprovidestheoverallruntimeexecutionenvironment.
Platformindependence
Letme putit this way- you could goand write anapplication inC++ for
Windowsthatimplementsawebservicethatlistenstoandrespondstorequests
andinteractswiththefilesystem.Butwhatwould youhavetodototake that
application and make it run on Linux? You would have to port it of course,
which requires a re-write of that C++ code to use operating system libraries
foundonLinuxmachines!
To get that to run on a Linux OS, you would port your code to use those
systemcalls that are availableon Linux. This image shows thatyou would be
writingyourcodeoverandoverforeachplatform.
Figure33-CodeportingacrossfromOStoOS
Nodeletsyouwriteyourapplicationcodeonce,andthenNodehandlesthe
lower level porting of Operating System level calls for you. Node.js acts to
abstractawaytheplatformOScapabilities.Notonlythat,butNodeallowsyou
towriteallyourcodeinJavaScript!
Figure34-Writingcodeonce
Node.js is an open-source project and has been ported to run on many
different operating systems. Much of the core code of Node.js is written in
C/C++ to enable native integration with underlying operating systems and
achievethefastestpossibleperformance.ItalsoutilizestheGoogleV8engineto
execute JavaScript. V8 actually compiles client JavaScript to native machine
code,suchasforx86machinearchitectures,forfasterexecution.
Node.jsiswell-suitedfornetwork-basedI/Oapplications.UsingNode,itis
extremelysimpletopiecetogetherawebserversimilartoIISorApache.You
can easily set up a web service to expose an HTTP/Rest API that works with
JSONpayloads.Nodereallyfulfillsalotofpurposesthatallowittosatisfyall
therequirementsofamiddle-tierservicelayer.
ExtensibilityofNode
TherealpowerofNodecomesthroughitsextensibility.Nodewaswrittento
providethecoreruntimeof execution,schedulingandnotificationcapabilities.
Itsfunctionality is then greatlyincreased through the manyextension modules
written for it. For example, there are modules for functionality such as
WebSockets, data caching, database accessing, asynchronous processing,
authentication,andmanyothers.
Nodeiswidelyadopted,andpeopleareconstantlyimprovingitandcreating
newmodulesforit.Thereisalargecommunityofdevelopersthathasgenerated
many extremely useful modules to give rich functionality to your application.
Since the patterns of using these modules are all very similar, it is extremely
easytoconsumethemwithoutanylearningcurve.
JavaScriptbliss
Since Node.js executes JavaScript, there is a consistency throughout the
applicationstackthatyouarebuilding.JavaScriptdesignpatternsareutilizedin
theconstructionofmodulesthatbecometheself-containedcomponentsthatare
availabletoconsume.Youcandownloadandintegratethoseofferedbyothers,
aswellascreateyourown.
There are also unit-testing frameworks that work with the JavaScript
languagethatyouwillusetotestyourNode.jsservicelayer.
JavaScriptisnowutilizedasmorethanjustaclient-sidescriptinglanguage.
Itisnowaformidableserver-sidelanguage,asimplementedwithNode.js.
Note:LestIgetaslewofemailsaccusingmeoflivinginafairytale,Iwill
make a brief comment on the sensibility of using JavaScript in enterprise
applications. It is true that JavaScript can be a challenge with respect to
deliveringonquality.However,withproperdesignpatternsandtesting,youcan
today create large enterprises services of superb quality with Node.js and
JavaScript. Many large companies have already done so. Perhaps as the
JavaScript language evolves more features will convince the skeptics that it is
heretostay.
8.3BasicConceptsofProgrammingNode
AsimpleNodeprogramthatyoucanwritewouldbeaprogramthatoutputs
texttotheterminalprocesswindow.Thefollowing isanexampleofwhatthis
wouldlooklike:
console.log("HelloWorld");
Youcantypethisintoafile,saveittodisk,andthenhavetheNodeprocess
execute it. If you had Node installed, you could open a command prompt and
typethefollowing,substitutingthenameofyourfilefor<filename>.
node<filename>.js
If you haven’t already done so, you should go ahead and install Node on
your machine at this time. Go to https://nodejs.org/. You will see a download
link labeled “LTS” and another labeled “Current”. You want the LTS version
(LongTermSupport),asthatisthestableone.Theotheroneisfromthelatest
codeunderdevelopmentandhasnotbeensufficientlyproven.
Go ahead and create a file named server.js, place the console.log("Hello
World");lineinitandrunnodeasshownwiththisfileasanargument.Youneed
tobeinthesamedirectoryasyourfile.
Youwillalsonoticethattheprocessdoesnotstayrunning.Inthiscase,once
yourcodeinyourfilehasbeenexecuted,theNodeprocessexits.Thisdoesnot
havetobethecase.IwillsoonexplainwhatwouldcauseaNodeapplicationto
keep running so it can continually perform server-side processing such as the
processingofwebrequests.
TheREPL
Node has several different options to control how it runs. If you were to
leaveofftheJavaScriptfile,NodewoulddefaulttowhatiscalledREPLmode.
REPL is the mode where you get a prompt and can enter JavaScript to be
executedasyoutypeitin.REPLstandsforReadEvaluatePrintLoopandisa
common thing for execution frameworks to provide. For example, MongoDB
providessomethingsimilar,calledtheMongoShell.
Inthisbook,youwillnotbeusingtheREPLandonlyneedtobeconcerned
with the main means of invoking Node as shown already, passing your
JavaScriptfileasanargument.
NodeexecutesJavaScript
Hereisanotherfiletotry.ThisoneillustratesabitmorecodethatNodecan
execute.Runningthiswillresultin“HITHERE343”beingdisplayed:
varx=7;
vars="Hithere";
functionblah(num,str){
if(num==0){
return"Can'tdothat";
}
returnstr.toUpperCase()+""+Math.pow(num,3);
}
varresult=blah(x,s);
console.log(result);
This illustrates some basic JavaScript language capabilities. If you look at
theJavaScriptspecification,youcouldseemoreofwhatispossible.Youhave
datatypes,operators,structuredprogramming,logiccontrol,built-inobjectsand
muchmore.
Ifyouhaveprogrammedbrowserscriptsbefore,becarefulaboutwhatyou
assume is available in JavaScript. For example, there is a function named
setTimeout()thatyoumightassumeisapartofthecorelanguageofJavaScript.
This is not the case. This is where browser implementations have added
functionalitytoJavaScript.
ThesetTimeout()functionisindeedprovided,butitisimplementedthrough
theNode.jslayersandnottheJavaScriptlanguageitself.Thisisalsotrueforthe
console.log()function.
As was mentioned, the Node process will run and exit upon execution of
yourlinesofcode.ThisisbecauseNode.jswillonlyrunwhileitknowsithas
codetoexecute.Tryrunningthefollowingcode:
while(true){
console.log("HelloWorld");
}
To stop the Node.js process, you will have to press <Ctrl> C on your
keyboard, or close the window. Later, you will see some code that requires
Node.jstorunforeverbecauseithasbeensetuptorespondtoeventsthatcould
perpetuallyhappen.
Usingbuilt-inmodules
As was explained, Node extends the basic capabilities of JavaScript by
providingasetofbuilt-inmodules.Therearequiteafewofthem,andyoucan
reviewthemifyougotohttps://nodejs.org/en/docs/.ClickonanAPIversionon
thelefttoseethelistofmodules.YouwillseethingsonthelistsuchasHTTP,
Net,OS,Crypto,FileSystem,Console,andTimers.
You can now see how to use the Node.js provided functions such as
console.log()andsetTimeout().Hereissomesamplecodethattakesadvantage
oftheseaddedcapabilities:
setTimeout(function(){
console.log('World!');
},1000)
console.log('Hello');
IfyouarenotfamiliarwithhowsetTimeout()works,youhavetobeaware
thatthisschedulesacallbackfunctiontorunsomenumberofmillisecondsinthe
future. Thus, the string Hello prints first and then one second later you see
World!.
Let’snowlookathoweasyitistocreateanHTTPWebserverwithcodethat
Nodewouldexecute. Thefollowingexample isallthe codeneeded ifyouuse
thebuilt-inHTTPmodule:
varhttp=require('http');
varserver=http.createServer(function(request,response){
response.writeHead(200,{"Content-Type":"text/plain"});
response.end("HelloWorld\n");
});
server.listen(3000);
IfyouweretoexecutethiswithNodeonyourlocalmachine,youcouldthen
openabrowserandnavigatetohttp://localhost:3000/andseeyourHelloWorld
messageappear.
The only non-obvious line of code is the very first one. This is a function
definedinNode.jsthatyoucalltoloadtheHTTPmodulethatprovidestheweb
servercapability.ThosewhowritebrowserscriptsinJavaScriptusean‘import’
statementinstead(seeNode10docsforusage).
Node uses the concept of modules as its extensibility mechanism. The
require()functionsimplyreturnsanobjectwithfunctionsandpropertiesplaced
on it that are specific to that module for you to use. In this case, you call
require('http')and get an object, and then use the createServer() function from
that object. Some things like the setTimeout() function are made available
globallywithoutyouneedingarequire()statement.
Usingexternalmodules
In some programming languages and runtimes, you mostly rely on what
comes built in. For example, this is the case with the combination of C# and
.Net.InNode,however,youwillconstantlybelookingforexternalmodulesto
add to your application to give you many of your capabilities. As a matter of
fact,sincetherearesomanyoftheseexternalmodulesprovidedonline,youwill
need to become good at searching for something and then determine what the
bestoptionis.
There is a package manager site you can go to for searching and
downloadingmodules.Gotohttps://www.npmjs.com/totakealook.Beaware
thatsomeofwhatisfoundinNPMarecodemodulesyoucanuse,whileother
downloadsareactuallytoolsyoucanrun,suchasthetestingtoolcalledmocha.
WhentheNPMmodulesareutilized,thelayersofcodenowlookasfollows:
Figure35-AddingNPMmodulestothearchitecture
Thisgivesyouthehighest-levelviewofthedifferentblocksofcode.
Note: Regular Node modules are written in JavaScript and distributed for
otherstouse.Thereis,however,awaytowriteamoduleinC++thatisreferred
to as an add-on. If you do that, you could get better performance (more
sustainedcomputeintensivecode)andalsoaccesstooperatingsystemAPIsthat
arenativetoamachinethatmightonlybeavailableinC++libraries.N-APIis
anewer capability providedby Node for buildingnative Add-ons. Learning to
create Addons is not necessary for most projects and the topic will not be
coveredinthisbook.
To use external modules from NPM, you need to have them installed
alongsideyourownJavaScriptfile.TherearetwostepsneededtouseanNPM
codemodule:
1. Createafilenamedpackage.json.
2. Runthecommand“npminstall<module>--save”foreachmodule
youwanttouse.
Running the npm install command will actually add a line in your
package.jsonfile. The veryfirst time you run itin a directory,a folder named
node_modules is created. If you look in that folder, you will see all of the
modulecode.WhenyourunNodewithcodethatrequiresoneofthesemodules,
Nodewillbeabletofindthem.
You can try the steps by creating the package.json file with the following
linesinit:
{
"name":"test",
"dependencies":{
}
}
Nowrunthefollowingcommand:
npminstallasync--save
Afterrunningthis,anode_modulesfolderiscreated,withtheasyncmodule
filesinstalled.Yourpackage.jsonfilewilllookasfollows:
{
"name":"test",
"dependencies":{
"async":"^1.5.2"
}
}
Thereisaconventioninthepackage.jsonfileforlistingmodules.Thename
islistedfollowedbytheversionyoudesire.Youcanspecifyanexactversion,or
you can specify just the major number and have NPM get the latest minor
version.The‘^’characterforourusageoftheasyncmodulemeansthatifany
timeyourefreshyourusagewithan“npminstall”commandyouwouldbringin
anyminoror patchupdates.For ourexampleabove, thatwouldnot bringin a
2.x.xversion,asthatwouldbeamajorversionupdate.
If you add dependency modules in your package.json file by hand and
specified the exact version, you could then run the “npm install” command.
NPMwillinstallthelatestversionforwhatversionnumberingyouspecify.The
followingcommandiswhatyourunifyoueditpackage.jsonfirstandthenwant
toinstallthemodulesyouspecified:
npminstall
Itistypicalfordeveloperstostickwithaknowngoodworkingversionfor
eachoftheirmodulesandnotupdatetoanynewmajorversionsevenwhenthey
become available. When you feel you need some new capability or security
patch of a newer version, then do an update and do extensive testing of
everythingagain.Majorversionshavenewfunctionality.
Hereisanothercodesampleyoucanrun.Itmakesuseoftheasyncmodule
thatwasinstalled.Putthiscodeinyourserver.jsfile:
varasync=require('async');
varfs=require('fs');
async.eachSeries(['package.json','server.js'],function(file,callback){
console.log('Readingfile'+file);
fs.readFile(file,'utf8',functionread(err,data){
console.log(data);
callback();
});
},function(err){
if(err){
console.log('Afilefailedtoload');
}else{
console.log('Allfileshavebeensuccessfullyread');
}
});
Youwilllearnmoreabouttheasyncmodulelater.Thisisbasicallyusingthe
async module capability to sequence through an array of values and do what
processingyouwantoneachentrysequentially.
Notethatyouareusingthefs module.Youdon’tneedto runaninstall,or
evenlistthefsmoduleinthe package.jsonfile.Thisisbecausethismodule is
partofNode.js,butyoustillneedtherequirestatement.Ifyounowrun"node
server.js",youwillseethecontentsofyourfilesprintedout.
Note: Deployment of your Node.js application is easy. You can just copy
everything,includingthenode_modulesdirectory,toamachine.Whenyouusea
PaaSenvironmentinstall,suchaswithAWSElasticBeanstalk,youdon’tneedto
copythenode_modulesdirectory.AWSwillrunthenpminstallfor you.Ifyou
want version numbers locked, set the specific versions or use what is called a
shrinkwrapfileoruseafeatureofNPM5forthat.
Callbacksandconcurrentprocessing
To start with, you need to understand that there is just one main thread of
executioninyourprogram.Thisisthethreadthatstartsupyourapplicationand
begins execution of your JavaScript code. From there, Node.js sends all your
codetotheV8JavaScriptVMengine,andOSportedcallstobeginitsexecution.
EverythingatthehighestlevelofyourJavaScriptplacesprocessingtimeon
asinglethread.Onlyonethingcanhappenthereatatime,sowritecodethatis
notcomputeintensive.
Note:VMstandsforVirtualMachineandisaconceptthatV8usestoisolate
JavaScriptexecution.Don’tconfusethisdefinitionofaVMwithaVMthatyou
findhostedinAWSorMicrosoftAzure.
YouhavealreadyseenNodecodethatusesthecallbackstyleofcoding.This
styleisprevalentineverythingyoudoinNode.TheNode.jslibraryprovidesfor
thesenon-blockingasynchronouscallbacks.Yourcodeneverblocks,butreturns
immediatelyandthenatsomelatertime,thecallbackfunctionisexecuted.This
givesyoutheconcurrentexecutioncapabilitythatNodeisfamousfor.
Note: There are other mechanisms for doing asynchronous code such as
usingPromisesandAsync/Await.Thesearealsoverypopularandarebasically
asyntaxpreference.Async/Awaitwillnotbecoveredinthisbook.
Codeexecutionflow
Thecodeexecutionpathisinterestingtotracethrough.Iwillnowexplaina
bitofhowthisallworks.Lookatthefollowingcodethatwillbeusedtohelp
understandtheexecutionflowforyourJavaScriptcode:
varx=7;
vars="Hithere";
varfs=require('fs');
functionblah(n,s){
if(n==0){return"No";}
returns.toUpperCase()+""+Math.pow(n,3);
}
console.log(blah(x,s));
fs.readFile('package.json','utf8',function(err,data){
console.log(data);
});
Ifthiscodeisinyourserver.jsfileandyouexecuteitonthecommandlineas
nodeserver.jsyourexecutionlooksasshowninfigure36.Youcanseethecode
boundarycrossings.Youcanalsoseethatalmosteverythingisnon-blocking.At
alowerlevel,thereisathreadinNodethatdoesendupbeingblocked,butthis
thread does not affect you at all. Your code still has the callback that is
asynchronouslycalled,soyouarenotblockedbyit.
Inthe illustration, the execution time moves leftto right. I have illustrated
theboundarybetweenyourmainJavaScriptthreadandtheNode.jsframework
with Libuv (see a later chapter for more information). The upper part is your
codebeingexecutedintheV8VM.
Figure36-Codeexecutionflow
Followlefttorightinthefigureaboveandyoucanseehoweachbitofcode
is run. To start with, the line of code that does the require() will block code
execution until it completes. The console.log() call does not block. The lower
layerwillasynchronouslyprintoutthevalue.Thefunctionblah()iscalledandin
that, the JavaScript Math.pow() object function executes synchronously in V8
andnotintheLibuvlayer.
Thefs.readFile()callusesanasynchronouscallbacktokeepyourupperlayer
codenon-blocking.YoucanseethethirdparametertothereadFile()functionisa
functioncallback.TheNode.jsframeworkstartsexecutingreadFile()forthefirst
bitofcode,buttheNode.jscodeisjustcallingintoLibuvtohandofftherequest
to be executed on its thread pool. This then immediately returns, and your
executioncontinuesinthecode.
TheLibuvexecution threadfor thefileI/O eventuallyreturnsand thenthe
callbackfunctiongetscalledandrunsonthemainthread.Youcannotgetaccess
tothethreadpoolprocessingdirectlyfromyourJavaScriptcode.
FilesystemcallsgotothethreadpoolofLibuv.Networkcallsareprocessed
differentlythanfilesystemcallsaswillbeexplainedlater.Ineithercase,Libuv
doesall the workfor you to makethings work acrossplatforms and in anon-
blockingway.
ContinuousprocessingwithNode
AnotherconceptthatyouneedtobeintroducedtoisthatofhowNodecan
berunninginacontinuousprocessingloop.Asshown,thepreviouscodesample
runstocompletionandthentheNodeprocessexits.ThisisbecauseNodeknows
ifthereisanymoreworktoexecuteandifthereisn’tany,itjustexits.
You can understand that there are certain modules you can use that will
basicallykeeptheNodeprocessrunningforever.Ifthisisthecase,youcanstop
theNodeprocessasyouwouldnormallydoonyouroperatingsystem.
Yousawcodeearlierthathadawhileloopthatneverhadanywaytoexit.
Thatexamplewouldbealittleodd,sinceitrunsallofthetime,andcompletely
blocksthesingleprocessingthread.Amorereasonablepieceofcodewouldbe
somethingthatusesanintervaltimertodosomeperiodiccomputation.Hereis
somesimplecodethatkeepsNoderunning:
setInterval(function(){
console.log('Helloagain!');
},5000)
console.log('HelloWorld!');
This is actually something similar to what the NewsWatcher code does to
periodically look for news stories. NewsWatcher will need to run some
processingonaperiodicbasis.
Anotherthingthatwillkeepyourprocessrunningforeverwouldbetheuse
ofmodulessuchasHTTP,NetorExpress.Whenyousetupthecodetolistenfor
TCP connections and listen on a socket, you set up code that will run in the
lowerlevelofLibuv.Takethefollowingexamplethatwasusedbefore:
varhttp=require('http');
varserver=http.createServer(function(request,response){
response.writeHead(200,{"Content-Type":"text/plain"});
response.end("HelloWorld\n");
});
server.listen(3000);
What happens here, is that Libuv is set up to use the low-level OS socket
capabilitiesto listenand respond to incoming connections and requests. Libuv
then has a loop to respond to any of these events. When they happen, your
callback code can run, such as the one above. The main Node.js process loop
actuallycheckswithLibuvtoseeifitneedstoberunningbecauseofworkithas
initsqueue,orislisteningfor.Ifso,thentheprocessiskeptalive.
Note: the Libuv thread pool is not involved in socket listening. This is
becausethelow-levelOScapabilitieshandletheasyncnon-blockingprocessing
andgeneratesthenotificationeventsuponcompletion.
8.4Node.jsModuleDesign
Node itself is composed of various modules that run as part of the core
service.ModulesarewhatenableallthefunctionalityinNode,besideswhatis
provided with the JavaScript language. The simple “Hello World” example
demonstrated this. That example made use of the console module. Other
modules, such as the express module, are third-party modules you install, to
bring in additional functionality. As mentioned, you use the Node Package
Manager(NPM)togetallyourexternalmodulesinstalledonyourmachineand
thenusearequire()statementtousethemincode.
You will write many of your own project code as modules for your own
consumption.Ifyouarereallyambitious,youmightevenwanttowriteamodule
and make it available as a download from the NPM repository for others to
benefitfrom.
I will now show you how a module is constructed, and you will see how
Nodeexposesthefunctionalityofamodule.Youdon’tnecessarilyneedtoknow
howthisworksinternaltoNode.js,butIincludethisinformationforthosethat
arecurious.
AmodulecanreturnaJavaScriptobject
Modules contain JavaScript code that is typically set up as an object. The
object is then exposed in a special way, so that a client can make use of it
throughtherequire()functionasshowninapreviousexample.Node.jsdoesthe
worktotakeyourmodulecodeandsurfaceitthroughitsinternalexposurefor
other code to call. Node keeps track of all the loaded modules and manages
loading, configuring, running, and caching of the modules. All require
statementsthroughoutyourcodewillreturnthesameobject(i.e.HTTP)asitis
cachedonceandusedforeveryrequire(singletonpattern).
Allyoureallyneedtoknowisthatmodulecodeisexposedthroughaspecial
Nodeobject named exports. The exports object is created for you by Node in
eachandeverymodulefile.Then,whencodecallstherequire()function,Node
returnstheexportsobjectwithwhateverfunctionalitywasplacedinit,suchasa
function. The following example shows a simple module you could write that
exposesasinglefunction:
//mymodule.jsthatholdsyourmodulecode
module.exports.welcome=function(name){
console.log(“Hi”+name);
}
AsImentioned,youprovidefunctionsandpropertiesinamodulefile.These
arethenexposedoutsideofthatfile.Youcanaddpropertiestotheexportsobject
suchasthewelcome()functioninthepreviousexample.WhatNodedidforyou
intheabovecodewastocreatetheexportsobjectwhenitingestedyourfile.
Node has code for the require() function that sets everything up to be
exposed. Node takes your code from your file, passed in as a parameter, and
doessomethinganalogoustothefollowing:
functionrequire(file){
module.exports={};
...Thefileparameterisusedandthatfileisopened
...andcodeisparsedandtakenandplacedbelow.
//Yourextractedcode
module.exports.welcome=function(name){
console.log(“Hi”+name);
}
//Endofyourextractedcode
returnmodule.exports;
}
Anemptyobjectiscreatedinthefirstlineofthefunction.Thatobjectthen
has properties added to it, such as the function you see. Finally, the object is
returnedsoothercodecancallthisfunctionofftheobject.
Forthecodetousethefunctioninthemodule,itneedstousetherequire()
function.Therequire()functiontakesthenameofthefilethathasyourmodule
codeinit.Youreferencemymodule.jsasfollows,andcallthefunctionyouhave
exposed.
//fileserver.jsthatusesthesamplemodule
varw=require(“./mymodule.js”);
w.welcome(“Bob”);
Nodeactuallyhasaninternalmoduleobjectthatitcreatesforeachmodule
exposed.Thereisalotmorethatisgoingonbehindthescenesthanthat,butyou
don’tneedtoreallyknowanymoreofthedetails.Youcancertainlylearnmore
of the internals if you are curious by reading through the actual source code,
sinceitisanopen-sourceproject.
Amorecomplicatedmodule
You can hang multiple properties on the exports object, such as objects,
strings,numbers,arrays,etc.Inoneoftheprevioussamples,Ishowedyouthe
useoftheHTTPmodule.IthasacreateServer()functionattachedtoit.Thisisa
designpatternknownasthefactory designpattern.Youdon’tusethefunction
directly,aswasdonewiththewelcome()functioninthepreviousexample,but
callittogetanobjectthatyoucanthenuse.
Youcanalsoexposeaclassthroughaconstructorfunctionthatclientstake
andconstructthemselves,oryoucangofurtherandprovideafunctionthatdoes
the creation for them like the factory pattern I just mentioned. If you have
multipleclassestoexpose,thenyouwouldwanttousethefactorypattern.Ifyou
onlyhaveoneclass,thenyoucanexposethatwithaconstructorfunction.
Ifyouwandtoprovideaconstructorfunctioninamodule,itwouldlookas
follows:
//mymodule.js
vara=require('http');
functionWelcome(nameIn){
this.name=nameIn;
}
Welcome.prototype={
this.name=null,
showName:function(){
console.log(“Hi”+name);
},
updateName:function(nameIn){
this.name=nameIn;
}
};
module.exports=Welcome;
Thefollowingexampleshowshowthismodulecanbeused:
varWelcome=require('./mymodule.js');
varw=newWelcome(“John”);
w.updateName(“blah”);
w.showName();
The welcome() function acts as the constructor in this example. The
prototype keyword in JavaScript allows you to set properties that exist for all
instancesandarethusnotcreatedagainforeveryinstance.Modulesaresingle
instance cached anyway. As Node keeps references to each that are used and
givesthesameinstancebackeverytimeitisrequired.
NotehowIincludedtheusageoftheHTTPmoduletobeusedbythesample
moduleabove.Thisshowshowmodulescanrequireothermodulesfortheirown
functionality and do so with the standard require() function. These included
modulesarenotvisibleoutsideofthatmodulecode.Thecodeusingthemodule
can’tactuallygetaccesstotheHTTPmodule,unlessitalsohadarequire('http')
statementaswell.
ImportinsteadofrequirewithNodeversion10
TheJavaScriptlanguageitselfhasreleases,andisanimplementationbased
on a standard called ECMAScript. Node has an implementation of JavaScript
thatisonapathtoadoptmoreECMAScriptsyntaxasthatevolves.IntheES6
release of ECMAScript, a new module system was introduced that uses the
‘import’keyword.Version10ofNodetakesasteptowardsimplementingthis,so
you would no longer need to use ‘require’ statements if you prefer to use the
‘import’ keyword instead. People writing HTML browser applications are
already familiar with this new standard. Here is what the old way looks like
comparedwiththenewerway:
//OlderCommonJSsyntaxthatexistedinNode
constsm=require(“./somemodule”)
//ESMsyntaxforgettingthedefaultexport
importsmfrom“./somemodule”
8.5UsefulNodeModules
Ifyougotohttps://nodejs.org/api/,youwillfindtheofficialdocumentation
onthecoreNode.jsmodules.Glancethroughwhatistheresoyoucankeepitin
mindifyouneedtoreferenceitinthefuture.
Besides the modules that come with Node, there are plenty of other
installable packages from NPM. https://www.npmjs.com/browse/star is a site
thatliststhemostpopularones.
Keepinmindthatmanyof theNPMdownloadsareforcode modulesyou
useinsideanapplicationandothersaretoolsthatyoudownloadandrun.Some
code modules are also for use as express middleware. Some are for other
frameworkssuchasReactorAngular.
Here is a very small list of some code modules you might find useful in
Node.js code that you write. A few come with Node itself and the rest you
downloadfromNPM.
Module: Purpose:
async
Usedtoforceyourcodetoruninaworkflow.Insteadof
doingthingslikenestingcallbacks,youcanuseasynctoset
upsequentialcalls.Alsoused torunparallel functionswith
theabilitytoknowwhenallhavefinished.
child_process Forchildprocessspawningandmanagement.
cluster
ForsettingupaclusterofNode.jsprocessestodistribute
theloadacross.
events
Aprimitivemodulethatmanyothersarebuilton,toemit
orlistentoevents.
express
Web server functionality for configuring routes, serving
up static files and providing template data binding
functionality.
fs StandardOSfilesystemfunctionality.
helmet
For mitigating different types of HTTP security
vulnerabilities.
http
Thismoduleservesadualpurpose.Youcanuseittoset
upalisteningserviceforincomingHTTPrequests.Youcan
alsouseittomakeoutgoingHTTPrequests.
joi
Forperforming validation onHTTP request JSON body
properties.
lodash
A collection of very useful helper functions (see
https://lodash.com/docs/)
mongodb UsedtointeractwithaMongoDBdatabase.
net
Standard low-level networking functionality for servers
andclients.
os BasicutilityfunctionsforaccessingOSinformation.
process
The standard type of functionality for working with
processesonanoperatingsystem.
request SimplifiesmakingHTTP/Sclientcalls.
response-
time
DisplaystheresponsetimeforHTTPrequests.
socket.io
A higher-level way to build bi-directional
communicationsbetweenclientsandservers.
stream
A primitive module that many others are built on to
providereadableandwritablestreamsontopofdata.
url UtilityfunctionsforworkingwithURLs.
util
InternalutilityfunctionsthatNodeitselftakesadvantage
ofthatareexposedforyoutouse.
zlib Forcompressionanddecompression.
Chapter9:Express
The E in the MERN acronym stands for Express. Express is a module
commonlyusedinaNode.jsapplication.ItishowevernotapartoftheNode.js
installation but can be downloaded from NPM and installed separately and
integratedintoyourapplication.Expressisverypopularandconsistentlyoneof
the most downloaded Node.js modules on NPM. The Express module allows
youtoimplementthefunctionalityofawebserver.Itprovidesawaytospecify
theroutehandlingforincomingrequestsandsimplifiesresponsegeneration.
OneofthethingsExpressdoesforyourprojectistoremovetheneedforthe
HTTP module that comes with Node. The HTTP module that is built into
Node.js requires you to write a lot of code in order to set up the routing and
responses.Expressmakesallofthateasier.
ThefollowingarethekeycapabilitiesofExpress:
SpecifytheroutehandlingofincomingHTTPrequests.
Mechanism to inject middleware into a request to modify it as it
getspassedalong.
Easy integration of third-party middleware to provide extended
capabilitiesforrequestprocessing.
Provides a requestobject with properties and methods to look at
everythingconnectedwiththeincomingrequest.
Provides a response object to use to set everything up for a
response.
ConfigurationforJSONpayloadserving.
Configurationtoserveupstaticfiles.
Pairsupwithserver-sidetemplateenginestosetcontextandreturn
HTMLwithdatabinding.
9.1TheExpressBasics
As with any external Node module, you have to do an NPN install and
requireExpressinyourcode.Youcandothisatthetopofafilesuchasyour
server.jsfile.YoucanthenmaketheExpresslisteneractive.Hereissomesimple
codetosetupanduseExpress:
varexpress=require(‘express’);
varapp=express();
app.listen(3000);
I will walk you through the construction of the NewsWatcher sample
applicationandshowyouhowtoseteverythingup.Beforethathappens,Iwill
coverthebasicsofhowExpressisused.
Expressconfigurationsettings
As part of using the Express module, you will need to configure some
settings in your code. These determine how it works and are settings you just
need at startup time. You use the Express object set()function to do this. The
app.set(name,value)syntaxisusedforsettingavalueforpredefinedvaluesthat
Express uses as configuration. The set()method will configure values such as
whatporttolistenon.Hereisanexample:
varexpress=require(‘express’);
varapp=express();
app.set(‘port’,3000);
To disable a setting, you can call app.set(name, false). You use the
app.get(name) method to retrieve any set value. There are enable and disable
functions, but they are the same thing as calling set with a true or false. The
followingsettingscanbeusedforconfiguringthemodeofoperationofExpress:
SettingswithValueoptions:
case
sensitive
routing
A Boolean to determine route interpretation as to case
sensitivity. For example, if set to false, then “/News” and
“/news”aretreatedthesame.
env
A string value that sets the environment mode such as
“development”.Thisispurelyforyourusetosetandread.For
example, you would have logic to determine which URL
endpoints, database connections, etc. depending on if you are
running the code to try out new development code, or if the
codeisrunninginproduction.
etag
UsedtosettheETagresponseheaderforallresponses.Set
ittostrong,weak,orfalseifyouwanttodisableit.Youcanalso
passinacustomfunction.Thedefaultvalueisweak.
jsonp
callbackname
AstringtospecifythedefaultJSONPcallbacknamesuchas
?callback=, which is the default. JSONP to bypass the cross-
domainpoliciesinwebbrowsers.Notnecessarilyneeded.
json
replacer
A string to specify the JSON replacer callback. This is a
functionyoudefinetodecideonaproperty-by-propertybasisif
theyarereturnedonaJSONrouteresponse.
json
spaces
Specifies the number of spaces to use for JSON indenting
forreadability.
port Theporttolistenon.
query
parser
Youcan setthis tosimpleorextended.simpleis based on
thequeryparserfromNodeandextendedisimplementedbythe
qsmoduleandisthedefault.
strict
routing
ABooleantosettotrueifyouwantrouteslike“/News”and
“/News/”treatedasdifferentpaths.Thedefaultsettingisfalse.
subdomain
offset
A number that defaults to 2 for how many dot-separated
partstoremovetoaccessthesubdomain.
trustproxy
Tobeusedifyouhaveafront-facingproxy.Youwillneed
tosetthisuptobetrusted.Thisisfalsebydefault.
views Thedirectory(s)tousefortemplatelookups.
view
cache
ABooleanthatenablestemplatecaching.Iftheenvsetting
issettoproduction,thenthisvaluedefaultstotrue,otherwise,it
defaultstofalse.
view
engine
A string to specify the template engine to use with your
providedtemplates.
x-
powered-by
A Boolean(defaults is true)to enable the"X-Powered-By:
Express"HTTPheadertobereturned.Youwouldsetittofalse
soasnottoreturnittopreventanyhackersfromknowingtoo
muchaboutyourimplementation.
TheExpressobjecthasapropertynamedlocalsonwhichyoucanplaceyour
own custom properties that you might want to associate with your Express
application.Youusetheapp.localsobjectasshowninthefollowingexample:
app.locals.emailForOps='Help@myapp.com';
Listening
Withcodeinplacetomanageyoursettings,allyouneedtodonextismakea
functioncalltoallowtheNode.jsruntimetorunitsplatform-specificcodeand
setupasocketconnectionforthegivenportontheservermachinetolistenon.
AnyincomingconnectionstotheIPaddressofthemachine,atthatspecifiedport
numberwillbeboundthroughtoyourcodetorespondto.Hereisthecodetodo
that:
varserver=app.listen(3000,function(){
debug('Expressserverlisteningonport'+server.address().port);
});
Withsome understanding ofthe initialization andsettings, you canlook at
implementingafullyfunctionalwebservicewithHTTPrequestroutehandling.
9.2ExpressRequestRouting
WhenanHTTPrequestcomesintoNode,therequestwillmakeitswayto
theExpress code you have writtento service it. YourNode.js instance willbe
runningonamachinethatishostedandexposedontheInternet.Nodewillbe
executinginaprocessonthatserver,andthroughtheExpresscodeitwouldbe
listeningonaportsocketforincomingconnections.
HTTPrequestscan cometo yourserverto rendera browserpageor fulfill
RESTwebserviceAPIrequeststhatdealwithJSONpayloads.AURLrequest
wouldbelikethefollowing:
http://newswatcherscale-env.us-west-2.elasticbeanstalk.com/news?region=USA
If you use query strings as shown above, you can get those values un the
Expressrequesthandlers.ThefollowingdiagramillustratesExpressrouting:
Figure37-Expressrouting
Ifyoureallyneedto,youcansetupthecodetoservicelowerlevelTCPor
UDPtypes ofconnections.Youcan researchtheNode ModulesNet andUDP.
ThischapterwillonlybeconcernedwithservicingHTTPrequestswithExpress.
Routes
OnceyouhavetheExpressJavaScriptobjectthroughtherequirestatement,
you can use methods that set up the servicing of HTTP requests. Express will
thenlistenforconnectionsonthespecifiedportanditcanunderstandthevarious
HTTPverbsandbreakdownwhat isbeingpassedinaspart oftheURLpath,
querystring,andtheHTTPheadersandbody.
TheExpressobjectgivesyoufunctionstouseforhandlingthevariousHTTP
verbs(get,putetc.).HerearethestandardverbsthatwouldbeusedforCRUD
typeoperationstoexposeanAPIinyourservicelayer.
app.get(path,callback);//Readitem(s)
app.post(...);//Createanewitem
app.put(...);//Replaceanitem
app.patch(...);//Updateanitem
app.delete(...);//Deleteanitem
functioncallback(req,res){
res.send(“Sendsomethingback”);
};
ExpresssupportsthefollowingHTTPmethods:
checkout
copy
delete
get
head
lock
merge
mkactivity
mkcol
move
m-search
notify
options
patch
post
purge
put
report
search
subscribe
trace
unlock
unsubscribe
Please see Express’s documentation for the complete list of supported
methods.Thegeneralsignaturelooksasfollows:
app.<METHOD>(path,callback[,callback...])
<METHOD>wouldbereplacedbyoneofthemethodssuchasget.Thefirst
parameteristheURLpath.ThisisnotthecompleteURL,buttheportionofit
afterthedomainname.
The second parameter is the callback function that gets called for that
request. You can actually provide multiple callbacks and they will each get
calledsequentially.Iwillcovermoreonthislater.
Routepathscanbestrings,stringpatterns,orregularexpressions.Theycan
alsobeanarraythatcombinesanyofthementionedformats.Asastring,aroute
pathcan be things like“/books”. Be aware thatthe query string portionis not
consideredpartofthepath.
Youcanuse“/”tospecifythatallpathsshouldbepickedup.Or,ifyouomit
thepathparametercompletely,allpathswillbepickedup.
Thecallbackfunctionyouprovidehasatleasttwoparametersinitsstandard
form.Thefirstparameteristherequestobject,whichcontainsinformationabout
theincomingrequest.Thesecondparameteristheresponseobject,andisused
to send back a response, such as serving up an HTML file or sending back a
JSONpayload.
ThefollowingexampleshowstheuseofapureRESTstyleURL.Youjust
needtoprovidetheroutingpaththatoccursafterthedomainportionoftheURL.
HereistheURLandthewayyouwouldspecifythepath.
//http://mysite.com/news/categories/sports
app.get(“/news/categories/sports”,callback);
For each verb, such as get, you might have different routes for different
resourcesthatarebeingretrieved.Hereissomecodethatsetsupmultiplepath
routesforthegetHTTPverb,eachreturningsomethingunique:
app.get(‘/about’,function(req,res){
res.send(“Aboutpage”);
});
app.get(‘/news’,function(req,res){
res.send(“Newspage”);
});
Be aware that the ordering of your route handling code is very important.
Any incoming request is basically consumed by the first path that is found to
handleit.
Asinglepathformultipleverbs
Ifyoufindyouhaveseveralverbsthatallrespondtothesamepath,youcan
specifythemtogetherbyusingtheroute()method.Thismighthelpyoualleviate
typinginthepathmultipletimes.Anexampleofthisis:
app.route('/customer')
.get(function(req,res){res.send('Getacustomer');})
.post(function(req,res){res.send('Addacustomer');})
.put(function(req,res){res.send('Updateacustomer');});
Allverbsatonce
Besidesthestandardverbs,therearealsothemethodsallorusetorespondto
allverbsoftheincomingrequests.Thefirstroutehandlerbelowisforallverbs
that are for the path /test. Then the second is set up for everything else and
returnsa404-NotFoundcode.Theusefunctionhereisnotusingapath,soitis
forallverbsandallpathsnotservicedyet.
app.all('/test',function(req,res){
resp.type('text/plain');
resp.send('Thisisatest.');
});
app.use(function(req,res){
res.type('text/plain');
res.status(404);
res.send('404-Notfound');
});
Advancedpathspecification
Sofar,youhaveseenthesimplestofcaseswithroutehandlingpaths.There
is a technique that can give you more advanced parsing capability for a URL
path.BelowisoneoftheURLsfromapreviousexample:
//http://mysite.com/news/categories/sports
app.get(“/news/categories/sports”,callback);
Theproblemwiththisexampleisthatyoumightneedtoservice20different
categories of news stories. For example, what about the paths
/news/categories/science or /news/categories/politics? It could get very
monotonoustosetuproutesforeachandeveryone.Tomakethiseasier,Express
allowsyoutosetupplaceholderparametersinthepaththatyoucanthengetat
laterinthecallbackcodeandthenhavethecodehandleit.
To be able to retrieve parts of a path, you use a special syntax in the path
parameter by placing a colon character in the string. This then sets up a
JavaScriptpropertyyoucanlateraccessinthehandlingfunctionaspartofthe
request object. The code below sets the path up with a colon. A property will
thenbeavailableontherequestobjectasshown.
//http://mysite.com/news/categories/sports
app.get(“/news/categories/:category',function(req,res){
console.log('Yourcategorywas'+req.params.category');
});
Withtheabovecode,youjusthavetheoneroutethatcanserviceallrequests
for news stories and can just feed that category into the backend retrieval
mechanism.
Thereq.paramspropertycanbeusedasanarray.InJavaScript,thatmeans
you can also access it as shown next. The following example shows this, and
alsoshowsthatyoucanhavemorethanoneoftheseparameterssetuptouse:
app.get('/products/:category/:id',function(req,res){
console.log(req.params[0]+req.params[1]);
});
YoucanalsoconstructyourURLssothattheycontainquerystrings.Fora
querystringthatyouwanttoprocesssuchas/news?category=sports,youdon’t
needanyspecialsyntaxinthepathparameter.Justspecifythepathuptothestart
ofthequerystringandstopthere.Thename-valuepairsofthequerystringwill
automatically be parsed out and made available to you as part of the query
property of the request object. For this example, there would be a property
namedcategory,withavalueofsports:
//http://mysite.com/news?category=sports
app.get('/news',function(req,res){
console.log(req.query.category);
});
9.3ExpressMiddleware
Atthispoint,youknowhowtosetupcallbackfunctionsthatgetexecuted
foragivenincomingrequest.NowI’llintroducetheconceptofmiddlewareasa
means of inserting route processing code that will happen before your ending
routehandler runs. Some middleware code will be provided by Express, other
middlewarecanbe by modulesdownloaded fromNPM, andother middleware
canfromwhatyouwrite.
Theconceptofmiddlewareisthatyoucanhavecoderunasaninsertedstep
thatisplacedbeforeyouractualroutecallbackgetsrun.Middlewarecodegets
chained together in a series of calls that you specify. Express Middleware can
thenactuponandmodifytherequestobjectthatisbeingpassedalongtheway.
Youcandothistoreusecodeacrossmultipleroutes.
Some inserted middleware terminates the request, as it takes care of
everything and the requests never even go to any of your route handling. An
example of this is the Express static file serving middleware. You might also
havesomemiddlewarethatinterceptscallstoverifyausersauthorizationand
doesnotcontinueiftherequestisdeterminedtobeinvalid.
Anotherexampleofmiddlewaremightbesomethingthatcachescontentfor
you.Anotherexamplewouldbeafunctionthatlogsalloperations.
Manyavailablemiddlewaremodulesaresimpletoadd,yetverypowerfulin
whattheyprovide.I’llshowyouseveraloftheminthischapter.Conceptually,
you can take the Express routing diagram shown previously and modify it as
follows to show middleware being injected that passes functionality down the
chain:
Figure38-Expressroutingwithmiddleware
Middlewarefunctionslookalmostidenticaltowhatyouhaveseenalreadyas
callbacks. They are just callback functions with the same signature you have
seenandthushaveaccesstotherequestandresponseobjects.Thismeansthat
therequestobjectcanbemodifiedbeforebeingpassedalong.Forexample,the
request body in the response object could be modified to have additional data
addedtoitbeforeitgetstoitsfinaldestination-handlingcallback.
You can read the Express documentation to learn about available modules
youcandownloadfromtheNPMrepository.Middlewarecanaccomplishthings
like authentication, caching, logging, session state, cookies, etc. These act as
sharedpiecesofcodethatyoucanuseacrossallorjustcertainroutes.
MiddlewareextendsthecapabilitiesofNode.jsbeyonditscorefunctionality.
Forexample,let’ssayyouarewritingawebserverthatwillserveupstaticfiles,
suchasimagefiles.Node.jsallowsyoutodothatifyouwritethecodetodoso.
However, there is a module that acts as middleware in Express that makes it
incrediblyeasytoimplement.Themodulethatdoesthisisthestaticmoduleand
itcomesaspartoftheExpressinstall.Itcanbehookeduptoserveupstaticfiles
withverylittlecode.
Note: Don’t confuse the concept of a middleware module with the general
concept of a module in Node.js. They are still provided through the requires
function,butareuseddifferentlythanregularNodemodules.
Hookingupyourowncustommiddleware
Normally,routeprocessingstopsatthefirstmatchthatisfoundforaURL
pathandthenacallbackisrun.However,ifyoumakeoneminormodificationto
yourcode,youcanstringtogethermultiplecallbacksforthesameroute.Notice
theonemodificationmadebelow:
//Firsthandlerfortheroute
app.get('/test',function(req,res,next){
...
console.log(“Gotherefirst”);
next();
});
//Secondhandlerforthesameroute
app.get('/test',function(req,res){
...
console.log(“Gotheresecond”);
});
Thedifferenceisthatthefirstroutecallbackfunctionhasathirdparameter
namednext. This parameter is a function, and its usage tells Express that you
wantthecallbacktoactasmiddlewarecodetobeinjectedbeforetheactualend
routeiscalled.Theorderisimportantasstatedbefore.
next()isafunctionthatyoucallwhenyourmiddlewarecodeisdonewithall
processing.Youmustcallnext()ortherequestwillbeabandonedandnotmake
ittoyourendhandler.Intheexample,thefirsthandlerrunsandthen,becauseof
thenext()functioncall,executioncontinuestothesecondhandler.
Thepreviousexamplecodecouldalsobestructuredsothatthecallbacksare
notseparatedout.Youthenjustlistoutcallbacksoneafteranother.Youwould
needtohavethecodeinthecb1()functionthatcallsnext()orcb2()willnotbe
run.
app.get('/test',cb1,cb2);
functioncb1(req,res,next){
console.log(“Gotherefirst”);
next();
});
functioncb2(req,res){
console.log(“Gotheresecond”)
});
Universalmiddleware
You can hook up middleware that will get inserted into every single route
and for every single verb. To do this, you simply use app.use(). This sets up
Expresstousethisfunctionacrossallincomingrequests.Youcanleaveoffthe
pathinthiscase,as“/”isthedefaultpathifyoudon’tprovideone.Ofcourse,
you may want to provide a path so that the middleware only gets run for a
certainpath.
app.use('/',function(req,resnext){
...
next();
});
Youcaninsertasmuchmiddlewareasyouneedforyourroutes.Theorderin
whichyoulisttheminyourcodewillbetheorderinwhichtheyaresequenced
through.Beawarethatcertainthird-partymiddlewarefromNPMarerequiredto
be placed before others. Refer to the middleware’s documentation for more
information.
I will now show you a practicalexample of some custom middleware you
mightwanttoimplement.Let’ssay thatyouhaveaspecialpath thatyouonly
want administrators to have access to. You can create a function that does
validationbeforeallowingtherequesttoproceedforfurtherprocessing.Hereis
howyoudothat:
//Middlewareinjection
app.all('/admin/*',doAuthentication);
app.get('/admin/stats',returnStats);
app.get('/admin/approval',approval);
With the app.all(), the authentication happens for all verbs and acts as
middleware. Inside the doAuthentication() would be code to determine the
authenticityoftherequest.Ifitwasdetectedtobeinvalid,thenyouwouldnot
callnext()andtheothertworoutehandlerswouldneverbecalled.Youwillsee
somethingsimilarbeingdoneintheNewsWatcherapplication.
If you call next() and pass an error object parameter, then that route
terminates from being handled normally. Error handling middleware then gets
invoked.Thistypeoferrorhandlingwillsoonbeexplained.
Parametermiddleware
Express allows you to set up a middleware callback function for a given
parameter property you defined in other route handlers. You do this with the
app.param()function.Thiscallbackiscalledbeforeanyroutehandler.Hereisan
example:
//Usingtheglobalparamhandler
app.param('id',function(req,res,next,value){
console.log(“someonequeriedid“+value);
if(value!=99)
next();
});
app.get('/products/:category/:id',function(req,res){
console.log(req.params[0]+req.params[1]);
});
Ifyoudon’thaveanyrouteswith“id”inthem,thentheparam()callbackwill
nevergetcalled.Theparamcallbackwillbecalledbeforeanyroutehandlerin
which the parameter occurs. You still need to call next() to continue the
processing.
Routerobject
If you really have a lot of routes and want to subdivide them for better
organization,youcanusetherouterobjectandkeepeachintheirownmodules.
Inthatway,youcanisolateyourlogicandnothaveitaffecttheotherroutesyou
havesetup.
You set the router objects up independently. Until you activate them with
app.use(),theywillnotbefunctional.Thefollowingisanexampleofsettingup
some middleware and some routes in for two different routers and then
activatingthemintheapp:
varrouterA=express.Router();
varrouterB=express.Router();
//Setuproutesthatwillendupwithnews
routerA.get('/weather',function(req,res){...
});
routerA.get('/sports',function(req,res){...
});
//Setuproutesthatwillendupforblog
routerB.get('/tech',function(req,res){...
});
routerB.get('/art',function(req,res){...
});
//Setsup/news/*
app.use('/news',routerA);
//Setsup/blog/*
app.use('/blog',routerB);
Middlewareerrorhandling
Anymiddlewarefunctioncanreturnanerror.Itdoessobycallingnext(err),
whereerris an Errorobject. Execution ofthe middleware and anysubsequent
routing is ended and processing of the error takes place. To process the error,
Expresshasadefaultfunctionthatitcallsthatwritestheerrorbacktotheclient.
Youhave the option of providing your own function or chain of functions for
handlingmiddlewareerrors.Ifyouprovideafunctionorchainoffunctions,then
thedefaultonewillnotbecalled.Inthefollowingexample,notehowthereare
fourparametersontheerrorhandlingmiddlewarefunction:
app.get('/test/:id',function(req,res,next){
if(req.params.id==0)
next(newError('NotFound'));
next();
};
//Amiddlewareerrorhandlingfunction.errisanerrorobject.
app.use(function(err,req,res,next){
console.error(err);
res.status(500).send('Somethingbadhappened');
});
Onceexecutionhasshiftedtotheerrorhandlingmiddleware,youcaneven
returnbacktoregularrouteprocessingifyoudoanext(‘route’)call.Doingthat
willjumpyoutowhateveristhenextdefinedroutehandler.
The diagram of Express routing can be further expanded to add in
middlewareerrorhandling.
Figure39-Expressroutingwithroutehandingandmiddlewareerrorhandling
Usingnext()evenifyourhandlerisnotmiddleware
I should clear one thing up for you though. Just because you add a next
parameter on a route handler does not mean you are implementing some
middleware.Forexample,youwillmostlikelyneednext()asaparameteronall
yourendroutehandlersinordertodocentralerrorprocessing.
As an example, the following code is an end route being handled by the
Expressrouter object. As it iswritten, you assume nothing willgo wrong and
justcarryoutsomeoperation:
varrouter=express.Router();
router.delete('/:id',function(req,res){
res.status(200).json({msg:'Loggedout'});
});
Whatifyouwantedtodetectanerrorandusemiddlewareerrorhandlingas
explainedintheprevioussection?Thisiswhereyouwillneedtoaddthenext
parameter to be called if there is an error. If there is no error, the route will
complete by calling the res.status()function to send back a response. Nothing
elsegetsintheway,andifrunningnormally,youdon’tcallnext()becausethere
isnothingelsetochain.
In order to use the error handling middleware, an end route itself must be
abletopasscontroltotheerrorhandlingchain.Todothat,thecodeneedstobe
modifiedtoaddinthenext()functiontobecalled.
Asmentioned,theendingroutehandlernevercallsnext()withoutgivingit
anerrorparameter.Thefollowingexampleshowstheadditionalerrorhandling.
Theerrorhandlercouldthensendaresponsewithanerrorcodeandmessage.
varrouter=express.Router();
router.delete('/:id',authHelper.checkAuth,function(req,res,next){
if(req.params.id!='77')
returnnext(newError('Invalidrequest'));
res.status(200).json({msg:'Loggedout'});
});
Staticfileservingmiddleware
One middleware module that comes with Express is the static middleware
thatallowsinterceptionofrequestsforfilesandreturnsthem.Thisalleviatesthe
needforyoutoprovideanendroutehandlerofyourown.Forexample,ifyou
wanttoserveupjpgimagefiles,youcanusethefollowingcode:
varexpress=require(‘express’);
varapp=express;
//Middlewareinjection
app.use('/images',express.static(‘images’));
app.listen(3000);
app.useissettingupthemiddlewareroute.Insteadofthecallbackfunction
beingprovidedbyyou,youinsertthecallforusingexpress.static().Thistakes
careofeverythingforyoutosendaresponseback.InyourNode.jsprojectcode,
youwouldneedtoprovidethefolderofimages.AnHTMLpagecouldaccessan
imageasfollows:
<imgsrc=”http://yousite.com/images/someimage.jpg”/>
Asasecondparametertoexpress.static(),youcanpassinanoptionsobject
that can have the following properties on it: dotfiles, etag, extensions,
fallthrough,immutable,index,lastModified,maxAge,redirect,setHeaders.
Forexample,setHeadersisafunctionyouwouldusetosetheaderstosend
with the files. Another example would be setting the lastModified property to
trueandtheLast-Modifiedheadervalueissettothedateofthefilebeingsent.
Formoreinformation,refertoExpress’sdocumentation.
Third-partymiddleware
There are a growing number of third-party modules that provide Express
middleware for your applications. Go to the Express site
(http://expressjs.com/resources/middleware.html) to find a list of modules you
candownload.
To use the middleware, you typically call app.use(<middleware>). This
meansthatallpathsandverbswillflowthroughit.Mostthird-partymiddleware
that intercepts routes before you get them will call the next() function so that
processingwilleventuallyreachyourcode,ifyouhavethatneed.
HereareafewusefulExpressmiddlewarecomponentsforyourreference:
passport
Usedtoauthenticaterequests.Youcansetthisuptologa
person in using OAuth (i.e. through Facebook), or federated
login using OpenID. There are more than 300 strategies
availablethroughthepassportmodule.
body-parser
Thismiddlewareinterceptsany HTTPPostverb requests
that have body data, such as from a form submit. The
middlewarecoderunsandthenbythetimeyourhandlercode
runs, the response object has a body property with sub-
propertiesoffofitforeachofthebodyvalues.
varbodyParser=require(‘body-parser’)
app.use(bodyParser());
//Inyourhandler,youcanlookatthevalues
app.post(‘/’,function(req,res){
console.log(req.body);
}
Youcan specifythat JSONis tobe parsedand placedin
thebody.Querystringvaluescanalsobeplacedintothebody
objectforyou.
varbodyParser=require(‘body-parser’)
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({extended:true}));
This will compress requests that pass through the
middleware.Youwouldplacethisstatementbeforeanyother
middlewareorroutes,unlessyouonlywantedcertainroutesto
be compressed. If you look at the headers of a returned
response, you would see the headers have an entry for
compression Content-Encoding.
varcompression=require('compression')
varexpress=require('express')
varapp=express()
app.use(compression())
cookie-
parser
This middleware does all the work to make available a
cookiepropertyonyourrequestobject.
varcookieParser=require(‘cookie-parser’)
app.use(cookieParser());
//lookatallofthepropertiesonbody
app.post(‘/’,function(req,res){
console.log(req.cookies);
}
errorhandler
Thisaccomplishesthesendingbackofstacktracestothe
clientwhenanerroroccurs.Makesuretoonlyusethiswhen
runninginadevelopmentenvironment.
varerrorhandler=require(‘errorhandler’)
app.use(errorhandler());
express-
session
Server-sidesessiondatastorage.
express-
simple-cdn
UsageofCDNfor staticassetservingwith multiplehost
support.
helmet
Helpful for mitigating several HTTP security
vulnerabilities.
response-
time
Response time tracking to add the X-Response-Time
header. The value inserted is in milliseconds. You could use
this to track your SLA over time and be alerted as to what
needsfurtherinvestigationforperformanceoptimization.
morgan
Thisisforrequestlogging.Thisfreesyouupfromwriting
anyofyourownconsole.logstatements.Forexample,incoming
HTTPrequestsgothroughthismiddlewareanditslogsthose
requests to the console window. You can also specify the
formatofthelogginganddirecttheoutputtoafile.
varfs=require('fs')
varmorgan=require('morgan')
//createawritestream(inappendmode)
varaccessLogStream=fs.createWriteStream(__dirname+'/access.log',
{flags:'a'})
app.use(morgan('combined',{stream:accessLogStream}))
multer Multi-partformdata,oruploadingfilesinchunks.
serve-
favicon
Forcustomizingtheiconinthebrowser.
timeout ForsettingatimeoutperiodforHTTPrequests.
express-
validator
Forvalidationofincomingdata.
connect-
redis
SessionstoreusingRediscache.
connect-
timeout
For routes that might run into some backend processing
issuesandneedtobelimitedintheamountoftimetheytake,
youcanusethistocutthemoffandreturnanerror.Youstill
needto determinewhat theright approachis for termination
andresubmissionofrequests.
varcto=require('connect-timeout')
app.get(‘/some_questionable_route’,cto(‘5s’),
function(req,res,next){
...somepossiblylongrunningcode...
...checkreq.timeouttoseeifitisevertrueand
...thenreturnfalse
returnnext();//finishedprocessingintime,gotonextfunction
},
function(req,res,next){
res.send(‘ok’);
}
);
vhost
For routing by hostname different sub-domains. i.e.
www.mysite.comversusapi.mysite.com.
express-
stormpath
Userstorage,authentication,authorization,SSO,anddata
security.WillworkwiththeOktaAPI.
9.4ExpressRequestObject
Let’slookmorein-depthattheusageoftherequestobjectinExpressroute
functionhandlers.Therequestobjectcontainsalltheinformationyouneedfor
digestingtheincomingrequest.Forexample,youhaveseenhowtheproperties
req.params and req.query are used. You have also seen some third-party
middlewarethataddsmorepropertiestotherequestobject.Hereisareferenceto
some of the properties that are available. You can refer to the Express
documentationtofindthecompletelist.
The request object is a parameter of your express route callback function.
Youcannameitanythingyoulike.“req”isagoodnameforit.Thefollowing
exampleshowshowtogetthecompleteURLthatthisrequestcameinfrom.
app.get(‘/’,function(req,res){
console.log(req.url);
});
Hereisareferencetothepropertiesthatareavailableontherequestobject.
RefertotheExpressdocumentationforthecompletelist.
Requestobjectproperties:
app
Areferencetotheinstanceoftheexpressapplicationobject.
params
This is used to access route parameters. You need to first
havesettheroutespecificationstringandthenyoucanusethe
paramspropertyoftherequestobject.
app.get('/user/:id',function(req,res){
res.send('user'+req.params.id);
});
query
Used to get the URL querystring. Aproperty will exist on
thequeryobjectforeach.
//ForURL“/users/search?q=Smith”,
app.get('/users/search',function(req,res){
console.log(req.query.q);
});
body
Containspropertiesofkey-valuepairsofdatasubmittedin
therequestbody.Youneedtoaddthebody-parsermiddleware
forittowork.
varbodyParser=require('body-parser');
app.use(bodyParser.json());
app.post('/',function(req,res){
console.log(req.body);})
route
Thisisanobjectthathaspropertiesofthecurrentroutesuch
aspath,keys,regexp,andparams.
cookies
Whenusing the cookie-parser middleware,this property is
anobjectthatcontainscookiessentbytherequest.Eachcookie
thatisattachedisapropertyonthecookiesobject.
req.cookies.someName
signedCookies
Exists if the cookies have been signed and protected from
tampering.
req.signedCookies.someName
ip
TheremoteIPaddressoftheincomingrequest.
protocol
Suchashttp,https,ortrustedifsetupwithatrustedproxy.
secure
ThishasavalueoftrueifSSLisineffect.
headers
TheHTTPheadersyoucanaccess.
req.headers['x-auth']
url
TheURLoftherequest.
path
Thepathpartoftherequest,withoutthequerystring.
route
Anobjectthatcontainsthematchedrouteandlotsofother
propertiessuchasthemethodandfunctionthathandledit.
hostname
ThehostfromtheHTTPheader.i.e.example.com
subdomains
The subdomain part that is in front of the hostname. i.e.
[“blah”,stuff”]iffromstuff.blah.example.com.
xhr
Thisissettotrueiftherequestcamefromaclientcallsuch
asfromXMLHttpRequest,whichhadsettheX-Requested-With
field.
Hereisareferencetosomeofthemethodsthatareavailableontherequest
object.RefertoExpress’sdocumentationforthecompletelist.
Requestobjectmethods:
get(field)
Togettherequestheaderfields.
req.get(‘content-type’);//i.e.“text/plain”
To check if a certain type is available, based on the
Acceptheaderfield. If whatyou sendin asa parameter
accepts(types)
doesnot match one of thevalues, then you will receive
anundefinedreturn.
req.accepts(‘html’);
is(type)
Tofindoutwhattypetheincomingrequestis.
req.is(‘text/html’);//i.e.returnstrue
acceptsLanguages(lang
[,...])
BasedontheAccept-Languagefieldoftheheader,it
returnsthefirstlanguageonamatch,orfalseifnoneare
accepted. Similar calls are acceptsCharsets and
acceptsEncodings.
varlang=req.acceptsLanguages('fr','es','en');
if(lang){
console.log('Thefirstacceptedis:'+lang);
}else{
console.log('Noneaccepted');
}
9.5ExpressResponseObject
Requestsareroutedtoyourcallbackbecauseofaroutingpathyouhaveset
up. Youwill eventually return a response back to the requester. The Response
objectiswhatyouusetodothatwith.YoushouldatleastsendbackanHTTP
status code. You might also return some HTML, text, or better yet, a JSON
payload.
Methodsontheresponseobjectarecombinedandhaveacumulativeeffect
onthereturnedresponse.Theexamplecodebelowsetsthestatusof200OKfor
asuccessfulHTTPrequest:
varexpress=require('express');
varapp=express();
app.get('/',function(req,res){
res.status(200);
res.set({‘Content-Type’:‘text/html’});
res.send(‘<html><body>Somebodytext</body></html>’);
});
app.get('/test_json',function(req,res){
res.status(200);
res.set(‘jsonspaces’,4);
res.json({name:’me’,age:37});
});
app.listen(3000);
Inthefirstroute,theContent-Typeissetas“text/html”andthenthesend()
methodisusedtofinishthereturnedresponsewithsomereturnedHTML.The
secondroutereturnssomeJSON.
Theresponseobjecthasmanyusefulpropertiesandmethods.Eachusageof
theresponseobjectisusedinsideafunctioncallback.Youcannameitanything
youlike.“res”isagoodnameforit.
Here is a reference to the properties and methods that are available on the
responseobject.RefertotheExpressdocumentationforthecompletelist.
Responseobjectproperties:
App
AreferencetotheExpressapplication.
headersSent
ABooleanvaluethatistrueifHTTPheadershavebeensent.
locals
Local variables scoped to the request, might be identical to
app.locals.Atemplatecanusetheseforitsdatabinding.
Responseobjectmethods:
status(code)
Usedtosetthestatusofareturninthecaseofanerror.
res.status(404);
accepts(types)
Forcontentnegotiationonareturn.
if(req.accepts('text/html')=='text/html')
res.send('<p>Hello</p>');
}elseif(req.accepts('application/json')=='application/json')
res.send({message:'Hello'});
}else{
res.status(406).send('NotAcceptable');
}
set(field[,
value])
Forsettingthefieldsoftheresponseheader.
res.set({‘contentType’:‘text/plain’,‘ETag’:‘123’});
get(field)
Retrieveswhatthesettingisforaheaderfield.
res.get(‘contentType’);
Thepathtoredirecttoinsteadoftheoneitcameinat.You
redirect([code,]
URL)
canprovideanoptionalstatuscode.A302“Found”isthe
defaultvalueofthecode.Youcanalsoredirectrelativetothe
currentURLoftheservice.
res.rediect(‘http://example.com’);
cookie(name,
value,[options])
Setsacookie.Thenameistheidentifierofthecookie.
Thevalueparametercanbeastringorobjectconvertedto
JSON.Theoptionsparametercansetupthingslikedomain,
expires,httpOnly,maxAge,path,secure,andsigned.
res.cookie('rememberthis','1',{maxAge:900000,secure:true});
json([body])
SendaJSONbodyback.
res.json({msg:'Hello'})
jsonp([body])
SendingJSONwithJSONPsupport.
res.jsonp({msg:'Hello'})
attachment([path
tofile])
Youcansetanattachmenttobereturned.Ifyoupassa
parameter,itisexpectedtobeafile.TheContent-Disposition
andContent-Typearesetforyou.
res.attachment('path/to/logo.png');
//Content-Disposition:attachment;filename="logo.png"
//Content-Type:image/png
sendFile(path[,
options][,fn])
Transfersafile.
res.sendFile(‘me.png’,{maxAge:1,root:’/views/’},function(err){});
end([data]
[,encoding]
Usethistoendtheresponsewithoutanydatabeing
returned.
res.status(404);
res.end();
format(object)
Usethisifyouaregoingtoreceiverequestsforcontentof
morethanonetype.Youcanlineupmultiplepiecesofcode
foreachAcceptHTTPheadertype.
res.format({
'text/plain':function(){
res.send('Hi');
},
'text/html':function(){
res.send('<p>Hi</p>');
},
'application/json':function(){
res.send({message:'Hi'});
},
'default':function(){
res.status(406).send('NotAcceptable');
}
});
append(field[,
value])
Addsthespecifiedtextandvaluetotheheader.
res.append('Warning','199Miscellaneouswarning');
send([body])
SendingofanHTTPresponse.
res.send({message:'Hello'});
sendStatus(code)
Setstheresponsecodeforthereturn.
res.sendStatus(200);
render(name,[,
data][,callback])
Templateresponsesending.Seethenextsectionofthis
bookformoreinformation.
res.render('user',{name:'Tobi'},function(err,html){
...
});
9.6TemplateResponseSending
OneofthethingsthatExpressenables,issendingserver-sideHTMLthathas
beengeneratedfromtemplatesthathavedataboundtothem.Thereareseveral
populartemplatelanguagesthataresimilartoHTMLmarkupthataresupported
throughExpress.Iwillhighlightjustone,butyoucaninvestigateothers.
Using the Express response object, you can formulate a response to send
backwithafunctionnamedres.render().Thisfunctiontakesafilethatcontains
thetemplateasoneparameter,referredtoasthe view.Asasecondparameter,
youcanprovidethedataobjectthatbindstothetemplate.Thetemplatethatyou
haveloadedthroughExpressbindsthedataandproducestheresultingHTMLas
theoutputtopassbackontherequest.
You can pass a third parameter as a callback function to get the rendered
string and process any errors that might have occurred. Here is what the code
lookslikethatutilizesatemplatetosendbackasaresponsetoarequest:
app.set('views',path.join(__dirname,'views'));
app.set('viewengine','jade');
app.get('/test',function(req,res){
res.render('test.jade',{
title:'MyNewsStories',
stories:items
});
});
First,youneedtohaveincorporatedtheNPMjademoduleintoyourproject.
ThenyouneedtomakecallstotellExpresswhatdirectorythetemplatefilesare
inandwhattemplateengineyouareusing.Expresswilltheninternallyusethe
Jademoduleyouhaveincludedinyourproject.
Noticethetwoapp.set()callsusedtoconfiguretheuseofJade.Theapp.get()
setsuptherequestroutewithahandler.Itisinthathandlerfunctionyouhave
thecalltorenderthetemplatewiththegivendatatobindtoit.
RefertoJade’sdocumentationto learnaboutthe templatesyntax. Itusesa
curlybracesyntaxtobindtoproperties.HereisaJadetemplatethatcouldtake
thepassedindatacontextandbindthosevaluestoelements:
//test.jadefilecontentforthetemplateview
doctypehtml
html
head
titlemyjadetemplate
body
h1Hello#{title}
div.newstbl
eachstoryinstories
div.storyrow
a(href=story.link)
img.story-img(src=story.imgUrl)
h6#{story.title}
Beawarethat,ifyouusetemplaterendering,youarerelyingonserver-side
renderingofyourHTML.IfyouprefertoutilizeaSPAarchitectureforaclient-
side browser application, you would not want to do this. In part three I will
describehowtoreturnHTMLthatusesReactcomponentDOMrenderingforthe
client-side SPA application. You can also do server-side rendering (SSR) with
React,eitherto returnall yourrendered pages, orjust afew ofthem. A React
Nativeapplicationwillalsobediscussed.
Chapter10:TheMongoDBModule
Thischapterisoneofthemostimportantonesinparttwoofthisbook.That
isbecauseoneofthemainpurposesofaservicelayeristoprovideaccesstothe
data layer. To do that, you will be utilizing a Node.js module that has been
createdtointeractwithMongoDBonthebackend.Youwillbelearninghowto
usethe“mongodb”modulefromtheNPMrepository.
Itisnecessarytofirstincludethismoduleinyourpackage.jsonfilesothatit
ismadeavailableinyourproject.Theusualrequire()statementisthenusedto
make functionality available in your code. I will cover that again when you
constructtheNewsWatchersampleapplication.
Note:ThemongodbNPMmoduleAPIisquitemassiveanditwouldtakea
large book to document it all. This book cannot make you an expert in all its
usage. For example, there are functions to create and delete collections and
perform other administrative duties that I have chosen to perform through the
AtlasmanagementportalandCompassapp.Youshouldcertainlymakeaquick
passthroughtheAPItoseewhatothercapabilitiesithasthatyoumightwantto
takeadvantageof.LookforthemongodbmoduleontheNPMsiteandfromthere
findalinktothedocumentation.
TheMongoClientobject
To begin with, your code needs to establish a connection to a MongoDB
server.Todothis,youusetheconnect()functionofthemongodbMongoClient
object. After establishing a connection, there are a lot of useful functions for
interacting with a MongoDB collection. The function signature for connect()
looksasfollows:
connect(urlConnectionString,[options],[callback])
The first parameter of the function is the URL of the service endpoint for
your MongoDB instance. The second parameter contains options that can be
usedforsettingsontheserver,suchasforareplicaset,etc.Thelastparameteris
yourcallbackfunction,whereyouwillreceivetheclientobjectthatthenallows
youtobeconnectedtoadatabaseandacollection.
IfyouhaveanincorrectURL,youwillgetanerrorreturned.Youcanfind
theURLconnectionstringtousebyopeningtheAtlasmanagementportal.Click
theCONNECT button for the cluster. Click Connect Your Application. You
willseetheconnectionstringlisted.YoucanclicktheCOPYbuttontocapture
it.URL-encodeyourusernameandpasswordiftheycontainanycharactersthat
arenotreadilyusedinaURLwithoutconvertingthem,suchas‘#’or‘@’.
Figure40-Atlasdatabasepagewithconnectionurl
Inmorereadabletext,thisissomethinglike:
mongodb+srv://test:pwd@cluster0-gas8f.mongodb.net/test?retryWrites=true
There are placeholders there for a password. You should also go to the
Securitytababovetheclusterandcreateauserloginthatyouwilluseinyour
code. This is different from the account login you use with the administrative
Atlasportal.Youneedtochooseasettingtogivetheuseraccountreadandwrite
accesswhenyoucreatetheuser.
Hereissomeexamplecodecallingconnect()toestablishaconnectionusing
theURLfromtheaboveexample.
vardb={};
varMongoClient=require('mongodb').MongoClient;
MongoClient.connect("<yourconnectionstring>",function(err,client){
db.collection=client.db('newswatcherdb').collection('newswatcher');
});
The code in the callback uses the database connection and calls the
collection()functiontogetthe“newswatcher”collectionobjectthatcanthenbe
usedtoperformCRUDoperations.
I can now show you a few of the methods exposed with the mongodb
module. The focus will be on learning the functions necessary to perform the
CRUDoperations.
10.1BasicCRUDOperations
With the collection object now obtained, you are ready to learn about the
CRUD operations. You can now learn about the four fundamental CRUD
operations necessary to utilize a MongoDB database. In the terminology of
MongoDB,forsingledocumentinteractions,CRUDtranslatestothefollowing
functions:
insertOne()
findOne()
findOneAndUpdate()
findOneAndDelete()
I’llnowwalkyouthrougheachofthefourfunctionsandshowyouhowto
usethem.Later,youwillputtogethertheNewsWatchersampleapplicationand
use them all again, plus a few others. At that time, you will make your code
morerobustwitherrorhandling.
Note:ThereareactuallyvariationsoftheCRUDfunctionslistedabove.For
example, to find multiple documents, it is necessary to use the find()function.
Whenreadinganyofthedocumentation,becarefultopayattentiontoanytext
statingthatitisdeprecated.
Create
The following is the function signature used for creating a document in a
collection:
insertOne(doc,[options],[callback])->{Promise}
Thefirstparameter istheJavaScript objectthat youwantto haveinserted.
This then gets created as a BSON document. If you do not provide the _id
property in your passed in JavaScript object, MongoDB will generate one for
youwhenitstoresthedocument.
Yourcallback function will have as the first parameter an error object that
youcanchecktoseeifsomethingwentwrongoncreation.Ifyoudon’tprovide
a callback function, a Promise is returned for you to use. The following code
passesanobject,andusesthecallbackfunction.
//Thisexampledoesnotusetheoptionsobjectparameter
db.collection.insertOne({property1:“Hi”,property2:77},function(err,result){
if(err)console.LogError(“Createerrorhappened”);
elseconsole.log(JSON.stringify(result.ops[0],null,4));
});
TheoptionsparameteroftheinsertOne()functionisanobjectthathasaset
ofpropertiesonit.ThissameobjectisusedformanyofthecallsintheAPI.I
willdescribeithereforyourreference,asyouwillseethisusedagain.Allthe
propertiesofthisobjectareoptional.Hereisatablethatdescribestheproperties
oftheoptionsobject:
RequestOptionsObjectProperties:
Property: Purpose:
w
The write concern (how the acknowledgment
works).
wtimeout
Thetimeoutyouwantforthewriteconcern.
j
Tospecifythejournalwriteconcern.
serializeFunctions
Booltoserializefunctionsonanyobject.
forceServerObjectId
Boolforserverassignmentof_idvaluesinsteadof
driver.
bypassDocumentValidation
To bypass schema validationin MongoDB 3.2 or
higher.
session
Optionalsessiontousefortheoperation
Theresultparameterinthecallbackfunctionhassomepropertiesyoumight
be interested in. One of them is the ops property. It was used in the previous
example and contained the returned document. The result properties are as
follows:
Property: Purpose:
insertedCount
Thenumberofdocumentsinserted.
ops
Anarrayofallthedocumentsinserted.
insertedId
ThegeneratedObjectId.
connection
Theconnectionobjectused.
result
ThecommandresultobjectreturnedfromMongoDB
Read
Toretrieveasingledocumentfromacollection,usethefindOne()function.
Thefollowingisthefunctionsignatureforretrievingadocument:
findOne(query,[options],[callback])->{Promise}
Thefirstparameteristhequerycriteria.Youcangobacktochapterthreeto
reviewwhatthatlookslike.Whatthisfunctiondoesistosimplyreturnthevery
firstdocumentthatmatchesthequerycriteriaandnomore.Thecallbackhasan
errorobjectfollowedbythedocumentreturned.Hereisanexample(notusing
theoptionsparameter)offindOne():
db.collection.findOne({email:"nb@abc.com"},function(err,doc){
if(err)console.LogError(err);
elseconsole.log(doc);
});
Optionsisanoptionalobjectwith20optionalpropertiesyoucanuse.Refer
tothemongodb module’s documentationfor acomplete listofproperties. The
followingisadescriptionofafew(therearemanymore)ofthem:
Name Type Description
fields
object
The projection criteria to specify the
fieldstoincludeorexclude.
hint
object Totellthequerywhatindexestouse.
explain
boolean
Return an object with query analysis
andnottheresult.
raw
boolean ReturntheBSON.
readPreference
ReadPreference
|string
Which machine in replica set to read
from,suchastheprimaryorsecondary.
maxTimeMS
Number
How long to wait in milliseconds
beforeabortingthequery.
session
ClientSession
Optional session to use for the
operation
findOne()returnsaJavaScriptpromiseifnocallbackwasprovided.
Youcanusetheexplainoptiontolookathowwellyourindexesareworking.
You will see some JSON returned that gives you useful data about the query.
Makesuretoremovethisoptionafterwardasitpreventsyouractualresultfrom
beingreturned.
Update
The following is the function signature for updating documents in
MongoDB:
findOneAndUpdate(filter,update,[options],[callback])->{Promise}
Thefirstparameteristhefilterparameterwhichisthequerycriterianeeded
toidentifythedocument.Thesecondparameterisfortheupdateoperatorstobe
used.Yourcallbackfunctionwillhaveasthefirstparameter,anerrorobjectthat
youcanchecktoseeifsomethingwentwrongontheupdate.Hereisanexample
usagethatusesanoptiontohavetheupdateddocumentreturnedinthecallback:
db.collection.findOneAndUpdate(
{email:"nb@abc.com"},
{$set:{name:"Charles"}},
{returnOriginal:false},
function(err,result){
if(err)console.log(err);
elseif(result.ok!=1)console.log(result);
elseconsole.log(result.value);
});
Thefollowingisadescriptionoftheoptionsobjectproperties:
Name Type Description
projection
object
Theprojectioncriteriatospecifythefields
toincludeorexcludeinthereturn.
sort
object
Species a sorting order for multiple
documentsthatarematched.
maxTimeMS
Number
How long to wait in milliseconds before
abortingthequery.
upsert
boolean Createthedocumentifitdidnotexist.
returnOriginal
boolean
Set this to false if you want the updated
documentreturned
Session
ClientSession Optionalsessiontousefortheoperation
findOneAndUpdate()returnsapromiseifnocallbackwasprovided.
Delete
The following is the function signature for deleting a document in
MongoDB:
findOneAndDelete(filter,[options],[callback])->{Promise}
Thefirstparameterisforthequerycriterianeededtoidentifythedocument.
Yourcallbackfunctionwillhave,asthefirstparameter,anerrorobjectthatyou
can check to see if something went wrong on deletion. Here is an example
usage:
db.collection.findOneAndDelete({email:"nb@abc.com"},function(err,result){
if(err)console.log(err);
elseif(result.ok!=1)console.log(result);
elseconsole.log("UserDeleted");
});
Hereisadescriptionoftheoptionsobjectproperties:
Name Type Description
projection
object
Theprojectioncriteriatospecifythefields
toincludeorexcludeinthereturn.
sort
object
Species a sorting order for multiple
documentsmatched.
maxTimeMS
Number
How long to wait (milliseconds) before
abortingthequery.
Session
ClientSession Optionalsessiontousefortheoperation
findOneAndDelete()returnsapromiseifnocallbackwasprovided.
10.2AggregationFunctionality
InthedatalayerchapterswheretheMongoDBcapabilitiesforqueryingwere
covered, I omitted one specialized type of query. What I omitted was the
capabilityofMongoDBtoperformaggregationoverthedata.Aggregationgives
youtheabilitytoreportonsummarizationsofdatasuchasgrouping,orfinding
thesum,min,ormaxacrossavalue.
This gets rather involved, so I left it until now to even mention this
capability.Youreallyneedtomakethisafocusofsomeseriousstudyinorderto
masterallthatispossiblewiththeaggregationcapability.
Hereisasimpleexampletogiveyouafeelforhowitworks.Imaginethat
for the example bookstore used in prior examples, you wanted to find out the
number of customers living in each state. You would use the aggregate()
function for this. The following example is the signature of the aggregate()
function:
aggregate(pipeline,[options],callback)->{null|AggregationCursor}
The pipeline parameter is an array of MongoDB supported aggregate
commands.Thinkoftheseasstagesthedataisbeingpipedthroughfromoneto
thenext.Therearequiteafewaggregateoperatorsyoucanstringtogether.
Hereisanexamplethatwouldgivethecountofpeoplebystate.Noticehow
wechainthefunctiontogotothetoArray()function.
db.collection.aggregate(
[
{$group:{"_id":"$address.state","count":{$sum:1}}}
]).toArray(function(err,result){
console.log(result);
});
Theresultmightbeasfollows:
[{_id:'UT',count:54},
{_id:'KS',count:988},
{_id:'FL',count:1259}]
Thecallbackhasanerrorobjectthatyoucancheck.
There are operators like $match and $project that you can insert to help
narrowdownthedocumentsandwhatpropertiesarepassedthroughthepipeline.
10.3WhatAboutanODM/ORM?
ThisbookshowshowtoconnectdirectlytoMongoDBwithanodemodule
createdspecificallyforthatpurpose.ButthereisanotherNPMmoduleyoucan
use that approaches interfacing with MongoDB in a completely different way.
Thatwouldbewiththe“mongoose”module.
Thosefromarelationaldatabase backgroundwillunderstandthatthere are
such things as Object Relational Mappings (ORMs) for connecting to a SQL
Server.Perhapsyou arefamiliarwithEntityFrameworkfor .Net,orHibernate
forJava?Theequivalentinadocument-baseddatabaseiscalledanObjectData
Mapping(ODM).
Lookup“mongoose”onNPMorGitHubandyouwillfindanODMthatsits
ontopofMongoDB.Thismodulecanbeusedinsteadofthemongodbonethat
wecoveredinthisbook.
The mongoose module adds an additional abstraction. With this, you get
featureslikeschematizationofthedataandvalidation.
Here is what code looks like that uses the mongoose module to save and
queryadocument.Iwilltakethesampleofthecustomerdocumentthatmight
exist in the online bookstore example. I will cut back on the number of
propertiesthough,soitisashorterexample.Hereiswhatsomecodewouldlook
likethatusesmongoose:
varmongoose=require('mongoose');
mongoose.connect('<TheusualconnectionURL>');
varCustomer=mongoose.model('Customer',{name:String,age:Number,email:String});
varc=newCustomer({name:'Aaron',age:32,email:'ab@blah.com'});
c.save(function(err){
if(err)console.log(err);
});
mongoose.model('Customer').find(function(err,customers){
console.log(customers);
});
There are ways to accomplish each of the ODM capabilities on your own.
For example, to add in a simple module that helps you do server-side input
validation,youcanuse“express-validator”or“joi”.Butwhygotoallthework,
ifanODMalreadyexists?AnODMmodulemaybethewaytogotogiveyou
morerobustnesswithyourapplication.
10.4ConcurrencyProblems
Onceyourapplicationstartstohavemultipleconcurrentusers,youmayrun
intoissuesupdatingdocumentsinMongoDB.Let’stakeanexamplewhereyou
haveasingledocumentthatcontainsalistofhighscoresforanonlinegameand
youwanttokeepthetopfivescoresacrossallplayers.Asusersfinishagame,a
callismade tosubmit thescore theyachieved andinsert itinto thelist oftop
scores. High scores are best, so if a new score is posted that should be added
becauseitishigherthanotherentries,thelowestscoreisdroppedoffthelist.
If multiple players all submit their scores at the same time, you can
understand that some contention might arise with updates to this single
document.Hereisadiagramofhowthismightlook:
Figure41-Topscorecontentionillustration
All of these users are causing score insertion code to run in parallel. Each
request gets sent to a Node.js Rest API endpoint route to allow multiple
simultaneous calls to read and update the single document in the MongoDB
DBMS.
Whatwouldhappeniftwoscoresaresubmittedforprocessingattheexact
sametime?Eachrequestwouldfirstreadthedocumentandtheninsertascoreif
itishigherthanthelowestnumberinthelist.Thedocumentwouldthenbesent
back for replacement in the collection. Here is a sequence diagram that
illustratestheproblemwithtimeflowingtoptobottom:
Figure42-Simultaneousupdate
If you look carefully, you can see that the final document is not what it
shouldbe.YouwouldwanttoseebothMegandJiminsertedatthetopandthen
KateandJoedroppedofffromthebottomofthearray.Instead,Megisatthetop
andonlyJoedroppedoff.Jimisreallygoingtobedisappointedwhenhechecks
thehighscoresanddoesnotseehisnamelisted.
Neither of the updates knew the other one was happening and so the last
write“wins”.ThisisaclassicproblemandnotsomethinguniquetoMongoDB.
There have indeed been many solutions invented over the years to solve the
problem.Forexample,somedatabase technologiesofferlocking.The problem
with this is that locking can be tricky in your own code and requires you to
implement detection of stale locks. MongoDB does not support API level
locking. It does, however, do this internally on its own for certain calls you
make.
ThesolutionthatMongoDBanditsunderlyingstorageengine(WiredTiger)
implement for you is termed optimistic concurrency. If you use the
findOneAndUpdate() function, this will happen at the document level. If you
were, however, to use read, followed by update functions, simultaneous reads
willstillgetthesamedataandonewritewillprevail.
With findOneAndUpdate(), if two calls come in, the first one pauses the
second one from doing anything until the first call has finished on that
document. The second call will not even do the read until the first one has
completeditswriteofthedata.Thesecondcallkeepsretryingforatimeuntilit
issuccessful.
Let’s go back to the initial sequence diagram and draw this out one more
timewithretriesputin:
Figure43-Simultaneouswriteswithoptimisticconcurrency
Optimistic concurrency retry logic can be inefficient if your database is
alwaysbeinglockedandretriesareconstantlyhappening.Imagineiffourusers
consistentlyupdatetheirscoresatthesametimeeverysecondoverandover.If
fourattemptsaremadeatthesametime,onlyonecansucceedandthentheother
threemusttryagain.Thesecondroundhasthreeattemptsandoneworks,andso
forth.
ThepointisthatyoumightbewastingMongoDBcomputetime.Thismay
causeyour MongoDB service to perform poorly. Youcan always look atyour
design and see if you really need to have one single document that is always
being contended for. You may run into cases where you simply have a “hot”
document with lots of simultaneous updates all of the time, as was illustrated
withthegamescores.
ImaginemultipleindependentNode.jsprocessesthatareallhittingasingle
backendMongoDBdatabase.Thismeansyouwillhavetocomeupwithaway
tocoordinateacrossindependentprocesses.
Ifyouwanttoreallyimplementtheultimatearchitecturetohandlemassive
scalingandavoidtheconflictsyougetwithoptimisticconcurrency,youneedto
createasinglequeuethatisoutsideofallyournodeprocesses.
Youwouldplaceyourupdateoperationrequestsinanexternalqueue,such
asanAWSSQSresourceforglobalaccessacrossallnodeprocesses.Youthen
need to create a single unique node process that would watch the AWS queue
andprocesstherequests.Thefollowingexampleshowsthissolution.Thedots
next to the Scaled Web API represents multiple machines with load balancing
acrossnodeprocesses:
Figure44-Queueserializationsolution
Thiswouldalsobethesafestsolutionifyoureallyhadtoensureconsistency
andrepeatability.Thereisonemaindrawbackwiththissolutionyouneedto
deal with the issue of returning to the initial caller if the operation succeeded.
Thismightnotbeimportant,butitcouldbe.
You can decide to take the “fire-and-forget” approach and always return
success. Perhaps, losing a top score for some reason can be considered
acceptableiflatertheworkerrolefailstoinsertascoreforsomereason.
Whatif,however,youhadabanktransactionthatrequiresanotificationone
wayortheotherbacktotheuser?Youwouldneedtoimplementsomewaytodo
an asynchronous return. There are notification mechanisms that are available
that would solve this. It is possible you could have some callback from the
queuingsystemthatcouldreturnifthetimewasinthemillisecondrangetonot
delaytoomuchtheHTTPrequestreturn.YoucoulduseWebSocketsorHTTP2
capabilitiestopushbacktothebrowsertheresult.Therequestwouldbequeued,
andanidreturnedthattheUIcanusetopollwithtogetthecompletionstatus.
Chapter11:AdvancedNodeConcepts
Therearesomeparticularlydifficultissuestobeawareofthatrequirecareful
coordinationacrosstheNodeecosystem.Therearecertainintricaciesthatmust
be handled properly. Done incorrectly, it can be disastrous. Done correctly,
everythingwillhum along.Thischapter willcovera fewof thesubtletiesyou
mightneedtoaddress.
11.1HowtoScheduleCodetoRun
Atimerfunctioncanbeusedtoschedulecodetorunatalatertime.Thiscan
be done as a one-time request, or it can be set up to happen on a recurring
interval.Youcanalsolookintousingthecronmoduletoscheduletherunningof
periodic Node code. The following code schedules a function to run in five
secondsandshowshowtopassinaparameter:
setTimeout(myFcn,5000,“five”);
functionmyFcn(param1){
console.log(“Hi%s”,param1);
}
Thecallbackisnotgoingtohappenexactlyat5000milliseconds,butNode
willdoitsbesttofititinwhenitistimeandthecallbackisthengiventoV8to
run.
YoucanusethefunctionsetInterval()toschedulearecurringfunction.Ifyou
want to cancel the recurring timer, you can cancel it at any time with
clearInterval().Thefollowingexampleshowshowthiswouldlook:
varid=setInterval(myFcn,5000,“five”);
...
clearInterval(id);
Be aware that Node will run your function over and over, even if the
previouscallhasbeenblockedandhasnotyetcompleted.Ifyouwantacallback
executed only if the previous one has completed, you could write your own
implementationofsetInterval()topreventthatbehavior.
Another function named setImmediate() is available that does not have a
timer go off, but places the callback to be executed after I/O, but before
setTimeout()andsetInterval()eventsareprocessed.
Ifyouareinterestedinschedulingcodetorunatanevenhigherpriority,you
canusetheprocessobjectnextTick()function.Becarefultonotusethisunless
you are using it responsibly. This call will place your callback to be executed
above all other processing in the event queue, even before I/O callbacks are
executed.Ifyoudothistoooften,atsomepointyouwillcompletelycutoutall
I/Oprocessing.
11.2HowtoBeRESTful
ItisuptoyoutodesigntheURLendpointsthatyourNodeapplicationwill
respond to. You can create a RESTful service and even support OData or
GraphQL if you like. Youare also free to include support for query strings, if
thatisyourpreference. Ifyou gothe RESTfulroute, thereare afew thingsto
keepinmind.
ThefirstthingtorememberisthatRESTfulwebservicesareintendedtobe
stateless.Ifyouarescalingyourserver-sideNodeprocessandusingthecluster
module, or have scaling through Elastic Beanstalk, then you most likely want
stateless servers. Any of your servers can process any incoming client call. If
you really want state information kept around, you can store state information
with the client, or have it cached for server-side retrieval in something like a
Rediscache.
Make sure to carefully design your REST API URLs so that they make
sense.Therearestandardthingstoconsidersuchasusingnounsandnotverbsin
yourpaths.Youmayrunintosomedilemmas,butthereisprobablyananswer
foryourchallengeinaforumsomewhere.
When you come out with a new version of your REST API, you need to
makethatapparentandperhapssupportmultipleversionsforawhile.Youcan
strive to keep your API as backward compatible as possible. You can insert a
versionnumberintoyourpathtomoveclientsfromoneversiontoanother.For
example, if you had “/api/users” as a path, you could have clients start using
“/api/v2/users”fornewfunctionality.Youcanalwaystackaquerystringonthe
endtospecifytheversioningsuchas“?Version=2015-12-22”.AnHTTPheader
settingisalsopossiblesuchas“x-version:2015-12-22”.
11.3HowtoSecureAccess
You should conduct a review of all your data connection points and
scrutinize all data that is being transferred and stored. Make sure to take
appropriate precautions with sensitive data you are safeguarding for your
customers.Informationsuchasahomeaddresscanbeusedtoidentifyaperson
and should never be leaked. Financial and medical records many times have
lawsandregulationsconcerningtheirstorageandtransmission.
You are not just trying to ensure your business interests are safe, you are
responsibletosafeguardyourcustomersfromanyharm.
One very important detail you need to work out is how users will identify
themselves and be allowed to access your Web API from a client application.
The other security concern you need to solve is how to prevent any
eavesdropping or man-in-the-middle type of hacks as information flows back
andforthfromtheclienttotheWebAPIservice.Thissectionwillexplorethese
andotherrelatedtopicsandpresentasolutionforeach.
Accesstoken
Once a person is identified by their logging in, a Web Service needs to
recognizethemandallowthemaccesstodataassociatedwiththeiraccount.One
possiblemeansofuserinteractionauthorizationistohavethemsigninandthen
useasessioncookiewitheveryrequestcomingin.
Another similar mechanism is to generate an access token and have that
passed in with every client request. There is a standard way to do this with
somethingcalled aJSON Web Token(JWT). A JWTis something thatcan be
generated on the server-side in response to a client login request when they
present a username and password. The JWT is basically an encoded set of
informationabouttheuser thatissignedtomake sureitis nottamperedwith.
HereisthesequencediagramforhowaJWTiscreatedandpassedfromlayerto
layer:
Figure45-JWTtokenpassing
You can pack into the JWT whatever you want. Put in information that
wouldhelpyouinyourprocessinggoingbackandforth.Onenicethingwould
betoaddintheIPaddressaswellastheHTTPuser-agentheadervalue.Thiscan
thenbevalidatedonlatercallstomakesureyoustillhavethesamepersonusing
thetoken.Youcanalsoset anexpirationtimeon thetokenso thataperson is
requiredtologineverysooften.
Tooffloaduserauthentication,youcanusetheOath2standard.Thisallows
youtohave aperson redirectedtosome otherwebpresence theyalready trust
and log in there first. Users might prefer this, as they only need to have one
singlesign-oncredentialtomanage.
WiththeNPMpassportmodule,youcanimplementOath2todelegatethe
userauthenticationtoanexternallytrustedsitesuchasGoogleorFacebook.A
tokenofauthenticityispassedbackthathasinformationaboutwhotheyare.
Note: The JWT should always be transferred using HTTPS because it can
easily be decrypted. Make sure it doesn’t contain anything that could
compromiseyoursecurity.Thetokenisintheheadersuchas“x-auth:<token>”
or“Authorization:Bearer<token>”ifyouwanttomimicstandardslikeOIDC.
Alldatatrafficshouldbeencrypted
Oneof the first things todo,once you have sufficientmomentumon your
code,is toimplement certificate-basedauthenticationand encryption usingthe
HTTPSstandard.Thiswillenableyoursitetobeviewedaslegitimateandalso
ensurethatdataistransferredusingtheencryptedTLS/SSLprotocol.
Ifyousearchontheinternet,youwillfindcodesamplesthatshowyouhow
to configure Express to require HTTPS. If your Node.js service were to be
hosted on a machine directly exposed on the Internet, this is what you would
needtodo:
consthttps=require('https');
constfs=require('fs');
constoptions={
key:fs.readFileSync('keys/agent-key.pem'),
cert:fs.readFileSync('keys/agent-cert.pem')
};
https.createServer(options,(req,res)=>{
res.writeHead(200);
res.end('helloworld\n');
}).listen(8000);
However, if you choose to use a PaaS solution, the above code is not
necessary.ThisisbecauseyourNode.jsserviceishiddenbehindtheserverthat
acts as the reverse proxy and load balancer. This means that the AWS Elastic
Beanstalk service external-facing load balancer needs to be configured for
HTTPS. You will see how this is set up when the sample application is put
together.
11.4HowtoMitigateAttacks
WhenyouexposeaserviceontheInternet,itwillbevulnerabletoattacksof
all kinds. Some attacks might be intentionally malicious and others just
annoying.Allthreatsshouldbetakenseriously.Ataminimum,theycandisrupt
your service, which is unacceptable. Beyond that, attacks can steal sensitive
informationanddodamagetoyourcustomersandtoyourownreputation.
Obviously,asimplenativemobilephonegameoftic-tac-toewouldnothave
as large an attack surface as a three-tier e-commerce application. The more
infrastructure and code that you have, the larger your attack surface will be.
Hackerswilllookforthevulnerabilitythatiseasiesttoexploit.Yoursecurityis
onlyasgoodasyourweakestpointofattack.
Toreallygetanaccuratelookatallpossibleavenuesofattack,youneedto
draw out a data flow diagram that shows all the processes, interactions, data
stores,anddataflows.Eachprocesswouldrepresentthosethatyouown,orones
thatyouarerelyingon.Someofthosecouldbeclassifiedascompletelyexternal
andpossiblyoutofyourhands,buttheyshouldstillbeonthediagram.
Foreachoftheelementsinthediagram,youwouldwanttodosomeanalysis
todeterminewhatthreatscouldexist.Forexample,ifyouhadaSQLdatabasein
yourdiagram,youwoulddeterminewhatitisstoring,howdatagetsinandout
and what configuration and administration are happening. You might discover
thatyourSQLdatabasewouldbevulnerabletoaSQLinjectionattack.
With all of your analysis done, you would mitigate each of the
vulnerabilities. In some cases, it might simply involve a few lines of code, in
other cases, you might need to re-architect parts of your system. To really
addressthistopic,youshouldbuyabookspecificallydedicatedtothattopic.I
willnowpresentafewsecurityconcernsassociatedwithNode.jsandWebAPI
interactions.
NevertrustANYinput!
Onebasicstrategytorememberistonevertrustanyinput.Alwaysdowhat
youcan to validate alldata before using it.Look into using thenode modules
“joi” or “validator”. Using a data layer ORM/ODM can give you these data
validation capabilities, but you still need to sanitize your data from script
injections.Hereissomevalidationusingthejoimodule:
varschema={
displayName:joi.string().alphanum().min(3).max(50).required(),
email:joi.string().email().min(7).max(50).required(),
password:joi.string().regex(/^[a-zA-Z0-9]{3,30}$/)
};
joi.validate(req.body,schema,function(err,value){
if(err)
returnnext(err);
});
You can see that only alphanumeric characters are allowed for the user
displayname.Thejoimoduleisalsomakingsurenoextrapropertiesexistonthe
body.Thesetypesofrestrictionsareextremelyimportant,sodon’tunderestimate
theirusefulness.
Another issue is data transmission size. What if someone started sending
reallylargeJSONpackages,oronesthathadextraobjectsorpropertiesinthem?
Youcansetupyourbody-parsermiddlewaretoturndownrequeststhataretoo
large. The default size is 100kb so you can make that smaller just to be safe.
Hereishowyousetthatup:
app.use(bodyParser.json({limit:'10kb'}));
Let’s now looks at some of the types of attacks that could occur on your
exposed Web API. None of these are really specific to Node.js. They exist
becauseofthefundamentalwaythatbrowsersandHTTPwork.
DOS/DDOSattack
Thedenial-of-service(DoS)ordistributeddenial-of-service(DDoS)attackis
where traffic is thrown at your web app to try to bring it down. It might be
possibletooverwhelmitsothatothersarepreventedfromusingit.Ifsuccessful,
the attack will deny service to the actual people that are intended to use it. In
somecases,itmightactuallyleadtoincorrectbehaviorofyourapplication,soas
toexploititforothergains.
The distributed version of the attack just means that the attacker is
employing multiple distributed machines at once. The term bot is commonly
used, meaning that these machines are set up to run scripts or programs that
carry out the attack and constantly enlist other machines to also participate. It
actslikeavirusandreplicates.
ADoSattackispurelymaliciousandwouldrarelyhappenjustbyaccident.
Regardless,youneedtobepreparedforitandmitigatethisrisk.Youcanbesure
thatbige-commercesiteslikeWalmart.com,Amazon,andeBayseethesekinds
of attacks and take them seriously. An attack like this might even cause your
scalinginfrastructuretokickinunnecessarilyandstartcostingyoumoremoney
incloudoperatingcosts.
I must, of course, bring up what has already been mentioned never do
anythingcomputeintensiveonthemainNode.jsthread.Thisisbecause,ifyou
havea lot of requests coming in that trigger someintensive synchronous code
thenyouwillhaveyourprocessbasicallyunabletorespond.Thiswilljustmake
iteasierforarealDoStooccurifyouletyourmainthreadgetoverloaded.Let’s
nowlookatwaystomitigateaDoSattack.
WhenyoucreateyourElasticBeanstalkNode.jsapplicationenvironmentin
the first place, it sets up Nginx to act as the reverse proxy and load balancer.
Nginx can be configured to limit the rate per IP address as well as limiting
connectioncountperIPaddress.
Another approach you can take using AWS is to set up an AWS API
Gatewayinfrontofyourservicelayer.Thatwillgiveyoualotofwhatyouneed
fordefense,suchasthrottlingperconnectiontoheadoffaDoSattack.Besides
that, you also get authorization, reporting, and API consumption of your web
APIcontractinacentralsharedway.
YoucouldalsodosometypeofIPblockingonyourown,suchastracking
the access time per IP and then limiting each IP to once per second access.
CheckouttheNPMmoduleexpress-rate-limit.Ofcourse,youcoulduseacloud-
mitigationproviderthatwouldusetheirexpertisetotrackpatternsofattackand
identifyDDoSattacksanddisablethem.
XSS-Cross-SiteScripting
AnXSShackiswheresomeforeignscriptgetsinjectedandrunas partof
yourweb-renderedsite.Alikelyvulnerabilitywouldbewhereyouareaccepting
input from the user and then later re-displaying that input back to them and
othersthatviewthesite.ThebrowserdoesnotbothertostopJavaScriptthatwas
maliciouslyputin,asitcannottellthedifference.Especially,ifyouconsiderthat
yourapplicationallowedtheusertoentersomethinginthefirstplace.Normal
userswillnotbetypinginmaliciousscriptstoberun,itisthehackersthatlove
todothisforfunandprofit.
Let’s take an example of something that could affect the NewsWatcher
application. In that application, people can comment on a shared news story.
This is an occasion where input from the user is accepted and later displayed
back to them. Let’s say that a malicious user enters the following text as a
commentonanewsstory,insteadofsomenicecomment:
<script>alert("Hi");</script><imgsrc="smiley.gif">
Nowallotheruserslookingatthecommentwillbeaffected.IntheUIcode,
youmighthavesomeHTMLthatdisplaysallofthecommentsandtheDOMas
follows:
<ul><li>
<p>'<script>alert("Hi");</script><imgsrc="smiley.gif">'</p>
</li></ul>
Everyonewillnowseeanalertboxandalsoacutelittlesmileyfacestaring
backatthem.Youneverintendedforthistohappen,butyoudidnothingtostop
it.Thesavvyhackercouldeven hacktheclient-sideJavaScriptcodeand mess
withtheJSONbeforeitgetssentbacktotheserver.WebAPIssimplycannever
trustthedatathatissenttothem
Imagine if the hacker referenced some script across the internet that really
wreaked havoc. If the hacker understood what your API was on the backend,
theycould runany commandas ifthey werea loggedin personand reallydo
somedamage.Thismeanstheywouldhavehijackedtheusersession.
Thisgoesbacktothesimplestatementthatyoushouldnevertrustuserinput.
Tomitigatethis,youcouldtakeactiontovalidatetheinputasyoucollectit.For
example,in the NewsWatcher code,you validate things likethe username and
don’tallow anythingbut alphanumericcharacters. Yoursanitizationcould also
scanallcharactersandchangeacharacterlike‘<’into“&lt;”.
Fortunately,fortheNewsWatcherapplication,whenyouareusingReactand
renderyourdata,Reactdoestheworktodisableanyscriptsfromrunning.React
simplydisplaystheactualtextwithoutlettingthebrowserinterpretit.Inthecase
of the example above, you would actually see the literal string
“<script>alert("Hi");</script><img src="smiley.gif">” in the comment list and
allwouldbegood.
CSRF-Cross-SiteRequestForgery
ACSRFhackiswherearequestismadetoasiteyouarecurrentlylogged
intowithyourbrowser.Youwouldhavealreadybeenauthenticatedandhadan
authorization cookie stored by the browser. The attacker would trick you into
viewingapagetheyhadsetupandasthatwasloadedinthebrowser,itwould
runa script thatwould send a requestto the siteyou had already beenlogged
onto.
Asanexample,let’ssayyouwereloggedontoyourbankingwebsite.Now,
whilestillloggedon,youopenupanemailthatwasfromamaliciousattacker
thatsaid:“Clickhereandwinamilliondollars!”.Whenyouclickonthelink,
thedestinationURLdirectsyoutotheirmalicioussitethatloadsapagethatruns
ascriptthatsendsrequeststoyourbankingsiteandtransfersmoneytothemand
changesyourpasswordatthesametime.Sinceyouarealreadyloggedon,the
browserhappilysendstheauthenticationcookiealongwiththerequestandyou
arehacked.Thebankingsitehadnoideathatthisrequestwasnotvalid.
Our NewsWatcher sample application does not use cookies, so it is not
vulnerable to this attack. The JWTtoken sending is under the control of your
clientcodeandisnotautomaticallysentbythebrowserlikeacookieis.
If you do end up implementing some design that is vulnerable to a CSRF
hack,youcanusethe“csrf”NPMmoduletoimplementamitigation.Thiswill
createasecrettokenthatonlyyoursiteknowsthatisonlysentfromyourpages.
TheNPMHelmetmodule
Ihavediscussedeachofthesecurityconcernsanddiscussedmitigations.In
thissection,youwilltakealookattheHelmetNPMmodulethatwouldgiveyou
theabilitytofurthermitigatepossibleattacks.
The Helmet module tweaks your HTTP headers to set things up to utilize
certainbestpracticesforsecurityriskmitigations.TheHelmetmoduleworksas
Express middleware by injecting itself into the request-response chain. It does
notdoanythingthatyoucan’tdobyhand.Irecommenditthough,asitwould
takeyoualotmorelinesofcodeforyoutoaccomplisheverythingthatitdoes
withasinglelineofcode.
IwillshowyousomecodethatwillbethestartingpointwhenusingHelmet.
ThiscodewillsetupthingslikeenforcingHTTPS,mitigatingclickjackattacks,
certainXSSmitigations,andattacksbasedonMIME-typeoverridingattacks.
Youcantakethedefaultsandfurtherspecifyanydeviationsfromtherethat
youlike.RefertothedocumentationforanyofthespecificHTTPheadersyou
wanttoindividuallycontrolonyour own.Thefollowingcodeshowsyou how
easyitistousehelmet:
varexpress=require('express');
varhelmet=require('helmet');
varapp=express();
app.use(helmet());//Takethedefaultstostartwith
Theoneusageyoudoneedtocontrolonyourownisthatwhichisusedwith
thesettingofaContentSecurityPolicy(CSP).Thisbasicallyletsthebrowserbe
aware of where resources can come from. This will then prevent resources
unknowntoyoufrombeinginjected.ThebasicusageofhelmetwithCSPadded
becomesthefollowing:
varexpress=require('express');
varhelmet=require('helmet');
varcsp=require('helmet-csp');
varapp=express();
app.use(helmet());//Takethedefaultstostartwith
app.use(csp({
//Specifydirectivesforcontentsources
directives:{
defaultSrc:["'self'"],
scriptSrc:["'self'","'unsafe-inline'",'ajax.googleapis.com',
'maxcdn.bootstrapcdn.com'],
styleSrc:["'self'","'unsafe-inline'",'maxcdn.bootstrapcdn.com'],
fontSrc:["'self'",'maxcdn.bootstrapcdn.com'],
imgSrc:['*']
//reportUri:'/report-violation',
}
}));
Ifyouwanttohaveviolationnotificationssentbacktoyourservice,youcan
uncomment the reportUri setting and then handle that Express route in your
code.Refertothedocumentationforsomesamplecodeforthat.Helmetcannot
be your only mitigation for security threats. You need a thorough analysis of
your data flow diagram to come up with every threat and start building your
securityplan.
TheNodeSecurityProjectinitiative
ThereisanongoinginitiativetoauditthecodeofNPMmodulesandkeepa
databaseofknownvulnerabilities.Thisisbynomeanscomprehensive,butitis
goodthatthisisunderway.StartingwithNPMversion6,anyinstallyoudowill
automaticallygetverifiedtotellyouifthereareanyknownvulnerabilitieswith
theNPMmodulesyouareusing.Youcanalsorunacommandtohaveanaudit
run:
npmaudit
Makesuretoexecuteitinthefolderwhereyourpackage.jsonfileis.
11.5UnderstandingNodeInternals
YounowknowenoughtouseNodewithoutknowingmoreaboutitsinternal
workings.Youarethusfreetoskipthissectionifyoulike.Youmay,however,
wanttocomebacktothissectionlaterasitmighthelpclearupsomeadvanced
questionsthatmightarise.
Note:ThissectionwaswrittenfromwhatIlearnedbyreadingthroughthe
actual source code for Node.js found on GitHub. Go to
https://github.com/nodejs/nodeifyou are interestedin the internalworkings of
Node.jslikeIwas.LibuvcodeisincludedintheNode.jssourcecode.Youcan
findoutmoreaboutitsAPIifyougotohttp://libuv.org/.
ThelayersofNode.js
You can see from the following block diagram that there exists a main
Node.jscodebasewithdependenciesbelowthat.Thetoptwolayersofcodeare
whatmakeuptheNode.jsframework.Thebottomtwolayersaredependencies
thatNode.jsrelieson.
The very top layer is the JavaScript library that you will be using directly
from your code. Any time your code does something that is outside of the
standardJavaScriptcalls,youwillendupusingsomethingthatisfoundinthis
layer. For example, all of the following code is made possible by the library
layer:
varhttp=require('http');
varserver=http.createServer(function(request,response){
response.writeHead(200,{"Content-Type":"text/plain"});
response.end("HelloWorld\n");
});
server.listen(3000);
This top layer is what is documented on the Node.js official site
https://nodejs.org/en/docs/asitsAPI.ToreallygetstartedanduseNode,thatis
the only layer you are really required to know anything about. All core Node
modulesareexposedthroughthislibrarylayer.
HerearethefourlayersthatcomprisetheoperationofNodeasaframework
runtime.Itispossiblethatthisdiagram mightchangeasNode.jschanges over
time.
Figure46-Node.jsplatform
ItwouldbegreatiftheOSitselfcouldunderstandJavaScript,butitdoesnot
havethelibrariesexposedforislikeitdoesforC/C++.Thus,thereneedstobe
some translation from your code to code that the operating system can
understand.Thisiswhythebindingslayerisneeded.
The Node.js C bindings layer is made up of C++ code. This is what takes
yourcodethatisinJavaScriptandallowsittocalldowntocodelibrarieslike
LibuvthatarewritteninC.
NodeusesV8asadependency.Itdoessofortwomainpurposes.V8makes
itpossibleforyourJavaScriptcodetocallthroughtoC++coderunninginthe
Node.js process. Take the following example of some Node.js JavaScript
application code you might have. This code displays the size in bytes of your
package.jsonfile:
varfs=require('fs');
fs.stat("package.json",function(error,stats){
console.log(stats.size);
});
ThewayitworksisthatNodehasexposedthefsmoduleinthelibrarylayer
thatisbeingusedhere.NodeusessomecapabilitiesofV8toactuallytakethe
fs.stat()call and have that make a call to a C++ function in the bindings layer
calledStat().ThisC++functioninthebindinglayerthenmakesacalltoLibuv,
whichinturncallOSappropriatelow-levelcode.Eventually,itmakesitswayto
aUnixflavorOSlibrarycallofstat()oronaWindowssystem,thecallmadeis
NtQueryInformationFile()thatisexposedin ntdll.dll.Thecallbackthenmakes
itswaybacktoyourJavaScriptwheretheasynchronousoperationcompletes.
V8actsasaVirtualMachineinthesensethatitcanisolateandexecutesome
codeandbehostedmanytimesononemachineindependently.Itprovidesallof
theaspectsnecessaryforalanguageruntime.YoucanreadanintroductiontoV8
JavaScriptengineby going tohttps://developers.google.com/v8/intro#about-v8.
Hereissometextfromthegooglesite:
“V8 is Google's open source, high-performance JavaScript engine. It is
writteninC++andisusedinGoogleChrome,Google'sopensourcebrowser…
V8 compiles and executes JavaScript source code, handles memory allocation
forobjects,andgarbagecollectsobjectsitnolongerneeds…V8does,however,
provide all the data types, operators, objects and functions specified in the
ECMAstandard.V8enablesanyC++applicationtoexposeitsownobjectsand
functionstoJavaScriptcode.”
BesidesLibuvandV8,thereareafewotherdependenciesthatNodeusesfor
operationsasnotedinthediagram.Toactuallylookatwhatthedependenciesare
ofNode.js,youcangototheGitHubprojectandlookinthedepsfolder.Node.js
isveryportableasitsdependencieshavebeenportedtomanyplatforms.
Youhavenowseenhowallofthelayersfittogether.YouseethatJavaScript
codecanbeexecutedandthatfunctioncallscanmaketheirwaythroughNode
modules.SomeofthosemodulesinvokecodethroughthebindinglayerinC++
downintoLibuv.
Thenextthingtounderstandishowtheprocessingloopcomesintoplay.I
leftoutexplainingthisthusfar,butitisimportanttounderstand.Understanding
theprocessingloophelpsyoutobeawareofwheretheprocessingtakesplace
andhowimportantitisforyoutokeepyourasynccallbackcodeasperformant
aspossiblesoastokeepthemainthreadfree.
OperationofNode
Do not believe everything you read about Node.js. Some people are under
the misconception that Node.js only has a single thread and can only do one
thingatatime.Thisissimplynottrue.Iwilldispelthatmisconceptionandteach
youexactlyhowNode.jsexecutes.
Nowthatyouhaveseenthelayersofcodethatyourapplicationsitsontop
of, you are ready to see how these layers actually operate to orchestrate the
executionflowofaNodeapplication.ThemainoperationofNodeisillustrated
in the following diagram. Not every dependant component has been included,
justthemainonesconcerningprocessingflow:
Figure47-Node.jsconceptualarchitecture
It is true that Node.js has a single-threaded architecture for its main
JavaScript processing. What is mostly executing on that thread are your
JavaScriptfunctionsthatNodeexecutesinresponsetoevents.Theseareinthe
formofasynchronouscallbacks.Thisiswhatgivesyouconcurrentprocessingin
yourapp.MeaningthatmorethanoneoperationisexecutinginthegutsofNode
andontheoperatingsysteminthebackground,butonlyonereturncanandbe
processedinyourcodeatatime.
Internally, Node can actually use multiple threads to shuttle work off to.
Node.js utilizes a callback code pattern which frees up the developer from
havingtomanagethethreadingandpollingthemselves.Forexample,youmake
your call for retrieving data from a backend database, and as part of that call,
you provide a callback function. Your code immediately returns and then
Node.js takes care of knowing when the data is returned and schedules your
callbacktorunatalatertime.Yourcallbackisthusrunasynchronously.
Node.jsdoes not have any sleep, mutex lock, or any similarfunctions that
youmightseeinotherruntimes.OnthelowerlayersofNode,suchasinLibuv,
thethreadsofNodecanbeexecutinginparallel,takingadvantageofrunningon
amulti-coremachine.TheOSwillbeabletotaketheexecutingthreadsofNode
and spread those out across the cores that are available. Thus, things like file
systemoperationsandnetworkrequestsdohappeninparallel,eventhoughyour
processingoftheresultscanonlyhappenserially.
Your JavaScript callback code is executed asynchronously, meaning you
makethecallanditisnotblockingandcodeexecutioncancontinueelsewhere
andyoudon’tknowwhenthereturnwillbeprocessed.Itisconcurrentbecause
yourJavaScirptcodecancallmanydifferentasynchronousAPIsandhavethem
allinaction,orqueuedwithoutyouworryingaboutthem.Itisparallelbecause
theOSwilltakethemanyprocessingthreads,orlow-levelcallsthatNodemakes
andspreadthemoutacrosstheavailableCPUcores.
Ifyou gobackand considerthe code sample that used the fs module, you
saw that it eventually made it all the way down to the lowest level OS stat()
function. That call is obviously a blocking call! This is where Libuv does the
work for you to queue up that request to take it off your thread and return
immediately. Libuv will then run the stat() call on a thread in its own thread
pool.Whenthecallfinishes,Libuvknowsthecallbacktocallandsendsitback
uptobeexecutedinV8.
The thread pool threads of Libuv are being used over and over. There are
fourofthembydefault,butyoucanchangethattobemoreifyoufindaneed.
The NewsWatcher sample does not use the filesystem directly, so it does not
alter that. The Net module ends up being used by the NewsWatcher database
interactionsandthatdoesnotusetheLibuvthreadpool.
BootstrappingofNode
If you look back at figure 47 you can see the upper left part has what is
termed the bootstrapping of Node. This is something that Node sequences
throughtogetupandrunning.Hereisthegeneralsequenceandexplanationof
thediagram:
1. The Node process is run from a command line and has a main()
functionthatiscalledasitsprocessentrypointinitsC++code.
1. Themainfunctioncreates theruntimeenvironment and
thenloadsit.
2. Anobjectcalled“process”iscreatedthathasproperties
andfunctionsonit.Thisobjectisveryimportantandisused
throughoutthecode.
3. A JavaScript file is now run in the V8 VM that
bootstrapsthewholeprocess.
1. Some internal Node modules are parsed and
madeavailable.
2. The file (i.e. server.js) that node was started
withasanargumentisreadandruninV8.
2. Yourserver.jsfilerunsandatthispointcanusethefullcapabilities
ofNodetodothingslikeuseJavaScript,setuptimers,setupHTTP
listeners,makeHTTPrequests,setupmiddlewareetc.
3. After your server.js code is finished being processed, the Node
processentersitsperpetualprocessingloop.Atthispoint,theprocess
keepsrunningaspreviouslystatedaslongasthereisworktoprocess.
1. Libuvprocessesworkonitsqueuewithitsprocessloop
andthreads.
2. Callbacksmakeitbacktothe mainthreadtoberunby
theV8VM.V8wouldqueuecallsthatcomeinandprocess
them.
Forthoseof youthatare stillskepticalabout mystatingthat therearetwo
processingloops,hereistheproof.Hereisthecodefromnode.ccthatthemain()
functioneventuallyentersandkeepsexecuting.Thisisreallywhatcanbecalled
themainprocessingloopofNode:
boolmore;
do{
v8::platform::PumpMessageLoop(default_platform,isolate);
more=uv_run(env->event_loop(),UV_RUN_ONCE);
if(more==false){
v8::platform::PumpMessageLoop(default_platform,isolate);
EmitBeforeExit(env);
//Emit`beforeExit`iftheloopbecamealiveeitherafteremitting
//event,orafterrunningsomecallbacks.
more=uv_loop_alive(env->event_loop());
if(uv_run(env->event_loop(),UV_RUN_NOWAIT)!=0)
more=true;
}
}while(more==true);
NotehowtheLibuvprocessingloopisnotallowedtocontinuallyrun,butis
calledandisunderthecontrolofthemainloopofthenodeprocess.Themain
Node process loop calls libuv to let it run with the UV_RUN_NOWAIT flag.
Themainloop thusiscontinuallycallingtohave theLibuvloop runoverand
over.HereistheUnixportedversionoftheLibuvprocessingloop:
intuv_run(uv_loop_t*loop,uv_run_modemode){
inttimeout;
intr;
intran_pending;
r=uv__loop_alive(loop);
if(!r)
uv__update_time(loop);
while(r!=0&&loop->stop_flag==0){
uv__update_time(loop);
uv__run_timers(loop);
ran_pending=uv__run_pending(loop);
uv__run_idle(loop);
uv__run_prepare(loop);
timeout=0;
if((mode==UV_RUN_ONCE&&!ran_pending)||mode==UV_RUN_DEFAULT)
timeout=uv_backend_timeout(loop);
uv__io_poll(loop,timeout);
uv__run_check(loop);
uv__run_closing_handles(loop);
if(mode==UV_RUN_ONCE){
uv__update_time(loop);
uv__run_timers(loop);
}
r=uv__loop_alive(loop);
if(mode==UV_RUN_ONCE||mode==UV_RUN_NOWAIT)
break;
}
/*Theifstatementletsgcccompileittoaconditionalstore.Avoids
*dirtyingacacheline.
*/
if(loop->stop_flag!=0)
loop->stop_flag=0;
returnr;
}
You can see that the while loop has code to cause a break statement to
happeninthecaseofNodecallingit.
Callback functions in your JavaScript usage of modules such as with
fs.reafFile() are kept by Node in a structure. When the low-level Libuv call
returns,thecallbackhappensontheC++mainthreadofNode.js.Ifyoulookat
thediagramagain,youcanseethereisatwo-wayarrowfromLibuvtotheV8
VM.
The call coming down from JavaScript through the C++ bindings uses the
V8 FunctionTemplate class to accomplish this. The callback going up from
Libuv makes it back to the C++ bindings layer code and uses the V8
Function::Call() function to have the V8 VM execute the actual JavaScript
callbackyouhadprovided.
ThiscallbackisnotexecutedontheNodeprocessthreadthatisorchestrating
all of this, but is executed in the V8 VM. Node is in no way managing any
queue,eventorprocessingloopfortheJavaScriptexecution.Thisisalldoneby
V8.
Theeffectofcomputeintensivecode
Themaineventthreadshouldonlybeusedtodofast,lessintenseprocessing
for inbound or outbound results. Take the following example that shows
processingthatisunacceptable:
varhttp=require('http');
varbcrypt=require('bcryptjs');
varcount=0;
varserver=http.createServer(function(request,response){
for(i=0;i<=10;i++){
bcrypt.hashSync("hjkl5678jhg",10);
}
response.writeHead(200,{"Content-Type":"text/plain"});
response.end("HelloWorld\n"+count++);
});
server.listen(3000);
Everywebrequest thatcomes inwill causethis compute-intensivecodeto
run.Thiswouldbasicallydevourallthecomputetimeasrequestskeepcoming
in.Youreventloopwouldnothaveanytimetoprocessanyincomingrequests.
The point here is that you should never do CPU intensive calculations in
yourmainNodethreadcallbackcode.Ifyoudo,itwillpreventtheentiresystem
fromworkingcorrectly.
Youcan solve this problem in a simple way with Node. There are built-in
capabilitiestosendexpensiveprocessingtoseparateforkedprocessesthatdonot
affectyourmainNode.jsprocess.Processingcanbeforkedtochildprocessesor
eventoaseparateexecutablerunningonyourcomputerthatyouhavewrittenin
anotherlanguage.YoucouldqueueupworkinacentralqueueandhaveNode
workerprocessespullfromthat.
Youcanalsocreateanadd-oninC++forNode.js.Youmightconsidersome
remoteservicecallsifyouhavecapabilitiesaccessiblethatway.Therearemany
creativethingsyoucandotosolvethisproblem.Asanotherpossibility,youcan
useAWSLambdatoaccomplishtheoffloadingofspotprocessingorotherwise
computeintensivecode.
In NewsWatcher we will be forking off code that does the fetching of the
global list of news stories. I only do it this way for now to simplify the
provisioning and deployment. The code that is running in the forked process,
couldbetakenoutandrunusingAWSLambda.
11.6HowtoScaleNode
CouldtheNode.jsprocessevergetoverloaded?Forexample,let’ssay1,000
HTTPGetrequestscametoyourNode.jsserverallatonce.Maybethis,inturn,
requires1,000filesbeingreadfromdisk.TheansweristhatNodeissetuptobe
abletomanageasizableworkload,soitmaydojustfine.Youwouldneedtodo
someprofilingforyourparticularworkload.
With few exceptions, Node.js makes your requests in a way that is non-
blocking.Forfilesysteminteractions,therequestsareshuttledofftothreadsin
theLibuvthreadpooltoruninparallel.Eachofthosedotheirworkandreturn
backtotheeventloopforcallbackexecutionscheduling.Asthreadpoolthreads
getfreeduptheyaregivenmorework.
Eventually,allofthe1,000filereadrequestswouldcomplete.Thishappens
intheleastamountoftimepossibleandalsowiththeleastamountofresources.
Allofthiscanbehandledquiteefficientlybyonesinglenodeprocess.
You can visualize this queue handoff as shown in the following diagram.
Note that the event loop of the Node process simply sequences through
everything in its queue if more come in than can be handled. You could
configurethroughcodetohavemorethanfourthreadsifyoulikeandyoucould
experimenttoseewhatperformanceimprovementsyoumighthave.
Figure48-Threadpoolhandoffforfilesystemrequests
Resultsarealwaysinterleavedandprocessedonebyoneonyoursinglemain
thread. As always, the performance is affected by how your JavaScript code
processestheresults.Thatiswherethebottlenecksalmostalwayshappen.
RememberthatnetworkI/Oishandleddifferentlythanfilesystemcallsand
doesnotusetheLibuvthreadpoolqueueaswasdiscussedforfilesystemusage.
AnyHTTP calls in your Node codeare handled by low level OSmechanisms
thatcanscale.
BeawarethattheNewsWatchersampleapplicationwillbeconnectingupto
aMongoDBdatabaseinthedatalayer.Thismoduleiswrittentouselow-level
TCPcallstotheMongoDBservice.Themoduleyougothroughtomakethose
calls uses the Net module of Node which eventually calls the library code of
Libuv to be making platform network calls. The MongoDB module pools
databaseconnections.Thismeansyouhaveacertainamountofconnectionsto
use for all the calls in your Node.js code and calls are queued and rotated
throughasconnectionsarefreedupfrompreviouscalls.Youcanconfigurethe
numberofconnectionsinthepool.
There is obviously a limit to how many operations you can achieve per
second before performance starts to degrade. Perhaps you have network I/O
requests going off to some slow-returning MongoDB call. The requests could
growoutofcontroliftheyarenothandledfastenoughandtheDBconnection
pool requests start to queue up. Of course, you also need to provide for
scalabilityinyourdatalayer,orthatwillbecomethebottleneck.
Note:At somepoint, Nodehas torelyon theOS andhardware(disks and
networkcards).Remember,thatyouhaveothersystemcontentionstobeaware
ofsuchasdiskcontrollercontentionsfromotherprocessesrunningonagiven
machine.
MultipleNodeprocessespermachine
Eventually, you can move to other architectural variations to handle your
load.Toscaletoagreatercapacity,youcanstartupmultipleNode.jsprocesses
on the same machine and distribute the load across those. If you have lots of
coresona machine,youcanmake useofall ofthemwith aNodeprocess for
eachcore.
TodothisscalingofNodeacrossthecoresofasinglemachineyoucanuse
the cluster module of Node.js itself to do the load balancing. There is also a
processmanagernamedPM2thatyoucandownloadfromNPMtodotheload
balancing. PM2 has other capabilities such as monitoring, restarting of Node
processes,andrunningrollingdeployments.
If you deploy your Node.js application through an AWS service, such as
ElasticBeanstalk,youcanutilizethepowerofPaaStoscaleeverythingforyou
byscalingupthenumberofcoresonanEC2VM(knownasverticalscaling)of
scaleupthenumberofEC2instances(knownashorizontalscaling).Eventually,
youwillnotbeabletohandleallrequestsandprocessingonasinglemachine.
Thsiswhenyoumuststartemployinghorizontalscaling.
SOAArchitecture
TheSOA(ServiceOrientedArchitecture)canbeunderstoodbycontrastingit
withwhat can beconsidered a monolithicarchitecture. Perhaps you haveseen
Java-basedorASP.Netwebservers.ThisiscodethatrenderstheHTMLonthe
server and sends it back to the client. It could have a lot of code that is
intertwinedtospitoutthisHTML.
Thistypeofmonolithicapplicationhasmanyshortcomings.Forexample,if
therewasachangetoonesinglepartofthecode,thewholeapplicationneedsto
bedeployedagain.Monolithiccodeistypicallytightlycoupledandveryfragile.
Themonolithendsupbeinghardtotest,enhanceandmaintain.
MovingtoaSOA,meansprovidingalayerthattheUIcouldaccesstogetat
functionalityasasetofservices.ThiscouldbeUIthatisserver-sideaswellas
from a client-side SPA. A great practice that came along was to create these
services as HTTP/REST web services that used JSON as the data transfer
format.
SOAwebservicesaresmallunitsofcodethatdoonethingandaretestable.
A Node.js application can use Express and have a nice RESTful approach
throughitsexposedpathsintheURLthatgoestotheroutehandlers.Thisends
upbeingaSOAwithallofthesebenefits.Yougetthebenefitsofsplittingout
thecapabilitiesintoseparatecodebases tobeseparatelydeveloped,testedand
maintained.Multiplepeoplecanworkoneachindividualserviceindependently
andeachroutehandlerisindependent.Eachservicehastheirownlogicandcan
go to their own database. They can actually go to the same database, but just
haveseparatedocumenttypesinsideofthatshareddatabase.
The code patterns for this type of SOA is what is implemented in the
NewsWatchersampleapplicationandcanachievesuperiorqualityandscaling.
You could deploy the same Node.js application across a horizontally scalable
cluster of machines. You could argue that you still have at least one of the
characteristicsof a monolith. Asingle codebase that has multiple services still
needs a complete deployment for each change that happens in its individual
services. The nice thing that makes it different is that each service really is
independentandisnotaffectedbyachangeinanotherservice,soitshouldnot
reallymatterifanewdeploymentofthecollectionofrouteshappens.
Scalinghorizontally
Let’s now discuss horizontal scaling. Perhaps you have four cores per
machineandhaveyourmainNodeprocessononecore.Youcancountonthe
OSmakinguseofallcoresforyouforoperationssuchasfilesystemoperations
thatusetheLibuvthreadpool.
Youcould employ the technique of forking processes from the main Node
process to offload some heavier computations and to do periodic batch
processing. If you are doing this, then you don’t have the spare cores to use
throughclusterorPM2usage.Thebetterapproachistospreadouthorizontally
acrossmachines.Youmustdothisanywayforreallyhighscaleneeds.
Scaling across machines is easy to configure in AWS. If you use Elastic
Beanstalk,itusesNginxasareverseproxyandloadbalancerforyourNode.js
application. Elastic Beanstalk lets you configure how many machines to load
balance between. You can also set up Elastic Beanstalk to use an auto-scaling
groupinsteadofusingindividualEC2instances.
Figure49-ElasticBeanstalkloadbalancing
EachEC2machinewouldalsobeinadifferentavailabilityzonetogiveyou
redundantfailoverifneeded.Thisisinadditiontotheloadbalancingbenefitthat
gives you more capacity. Figure 49 shows what that looks like with two EC2
VMinstances.
One nice thing with AWS Elastic Beanstalk, is that when machine OS
upgradesorpatchesneedtohappen,itisdoneonemachineatatime.Machines
are taken out of rotation, updated and then put back into rotation. When you
needtoupdateyourNode.js application,yourdeploymentalsohappens inthis
exactsamewaywitha“rollingupdate”.Thiswayyouachievewhatiscalleda
“zerodowntime”deployment.
Note:Beawarethatwithanyofthesescalingmechanisms,youmustensure
thatyouarerunninginacompletelystatelesswayandavoidanyaffinitysettings
to truly be able to distribute calls across all Node processes. To manage a
connectionrequiringstate,youcaninserttheuseofsomethinglikeRedistofetch
stateifneeded,orkeepstateonlyontheclientorinatokenthatispassedback
andforth.
Staticresourceserving
IntheNewsWatchersampleapplication,theNode.jsapplicationactstoserve
uptheHTTPrequestsfortheAPIroutes.ThesearetheRESTfulwebservices
thattheReactUIconsumes.TheNode.jsapplicationservesuptheactualReact
website.Thesearewhatarecalledstaticresources,becausetheydon’tchange.
All files, such as images, css, html etc. required for the React application are
containedinacompletelydifferentdirectorythatNodeknowsaboutandserves
up.Forexample,hereistheloggingthathappenswhentheReactapplicationis
requested.YoucanseetheJavaScriptbundledfileforReactthatisbeingsent.
The 304 code you see means that a resource was not send since the version
requestedwasstillthelatest.
Figure50-Staticresourcereturns
TherearemoreefficientwaystoserveuptheReactapplicationandfreeup
theNode.js application fromthat work. Forexample, the Nginx reverseproxy
load balancer can actually know about those requests and serve them up.
Another way would be to use something like Amazon S3 storage and set up
AWSRoute53togothroughAWSCloudFrontthatpullstheReactapplication
from the S3 storage. This is what is referred to as a CDN (Content Delivery
Network).
In the case of Nginx or CloudFront, they both support caching. Since the
React application resources are not updated often, requests would be fetched
fromthecacheandbeextremelyfast.Express.staticdoesnotdoanycaching,but
you can use client-side caching with ETag or Max-Age. By default, the static
middlewarehasETagsenabled.
ADockerContainerforyourNode.jsprocess
It is possible to use Docker as a means of deploying and running your
Node.js application. Elastic Beanstalk can even use a self-contained Docker
container.Yougetthesameprovisioning,loadbalancingandscalingyougetby
justdeployingastraightNode.jsapplication.
Youmightwanttotakethisapproachifyoufindthatyouhaveotherthings
to install alongside the Node.js application that can all be contained in the
Dockercontainer. If allyou have isyour Node.jscode to deployusing Elastic
Beanstalk,youmightnothaveanybenefitinusingDocker.
Itcanbeconvenientfordeveloperstobeworkinginthiswaysothatthereis
some structure and predictability as to what is needed for a deployment. This
wouldmeanthatifyouhaveaDockercontainerallrunningandtestedonyour
developmentmachine,itismorelikelytoalsofunctioninproduction,oratleast
notbemissinganythingthatneedstobeinstalledalongsideit.
Note:You can keep your Node.js application Docker images in AWS ECR
(EC2ContainerRegistry).
MicroservicesArchitecture
We discussed that implementing SOA was a great idea and that we have
achievedthatwithourRESTfulapproach.Aswasmentioned,theNewsWatcher
sample application keeps all of the RESTful routes in the same application.
ThinkaboutwhatwouldhappenifeachindividualHTTP/RESTroute(i.e.auth,
user,etc.)weresplitintotheirownNode.jsapplicationanddeployedseparately.
This gets us into the topic of realizing an architecture that is called a
Microservices architecture. The following figure shows the before and after
visualization of how our NewsWatcher application could be made into a
Microservicesarchitecture.
Figure51-Microservicesclusterdeploymenthosting
Each different background fill pattern in the diagram would represent a
differenthostedserviceendpoint.Forexample,Billing,UserProfile,Inventory
and such. Each can be independently scaled and the load is balanced across
availablemachines inthe cluster. Amazonhas ECS forthis andalso EKS and
Fargate.
Microservices are independently deployed and run and each has their own
API,middlebusinesslogicandbackenddatalayerasnecessary.Thedatacould
bedocumentsinasharedMongoDBcollection.Microservicescanbethoughtof
asaspecificwaytoimplementSOA.
ThetermSOAcertainlysurfacedmanyyearsagoanditsimplementationsat
thattimewereveryformalandburdenedbyspecificationssuchasSOAPWS-*,
XML/WSDLand were sometimes implemented ontopof an ESB(Enterprise
ServiceBus).Thatevolvedovertime,buttheoriginalconceptsarestillvalidin
today’smodernHTTPRESTandJSON.
Before rushing into a Microservices architecture, you should be sure you
reallyneedoneinthefirstplace.Implementingafullbuildanddeploysystem
formicroservicesismorecomplicated.Alwayskeepthingsassimpleaspossible
foryourneeds.Let’s lookatafewreasons peoplegiveforwhyMicroservices
aresogreatandunderstandunderwhatcircumstancesyoumightwanttoattempt
this.
One benefit is that services are split out into discreet pieces that are much
smaller.Theargumentisthatsmallerpiecesareeasiertobuildandtest.Youmay
haveheardofcompaniesthatmanagethousandsofmicroserviceatonce.Small
Microservicesprovideallofthebackendservicingforanynumberofconsumers
on the front end. Amazon is one such company that utilizes Microservices.
Chances are that you are a much smaller company than Amazon and your
applicationsaremuchsimpler.
Remembertodowhatisrightforyourneeds.Buildyourbackendservicesat
a granularity that makes development and management easiest for your
operation.ItisunfortunatethatthetermMicroservicesstartswith“micro”.This
makesone thinkthatthey mustbe really small.Indeed, some peopleadvocate
microservices that are 100 lines or less. If you really want to do that, then I
would advise you go to a serverless architecture and not really do Node.js
application,buttolookatthingslikeAWSLambdaFunctions.
Ifyouwerewritingabackendbillingservice,youcanimaginethatitcould
be a large amount of code. You can initially try to split it out into smaller
consumableunitsthatcouldeachbeusedindependently.Iftheycannotbeused
independently,thenitshouldbeonecohesiveservice.Atonepoint,astatement
was made in a presentation by Amazon on Microservices that the main
amazon.com page makes 100 to 150 backend calls to Microservices at initial
loadtime.Theseareverysmallconsumableservices.
Anotherbenefit that is always broughtup is that you can scalebetter with
microservices.Thatisofcoursedebatable.IfyouweretotaketheNewsWatcher
applicationasitisanddeploythatonafewdozenmachines,youcouldscaleto
tensofmillionsofcallsaday.Whatwouldhappenifyousplitouttheroutesinto
theirowndeployableNode.jsapplicationservicesanddeployedthoseacrossthe
same machines? You might actually get poorer performance because each
machine would be running a set of Node.js processes instead of just one.
Obviously, the argument is that not all services would be deployed across all
machines in the same amounts. Services like billing, might not get called as
oftenandcouldbedeployedonafewmachines,whileaproductlookupservice
mightbecalledalotmoreoftenandneedtobedeployedontolotsofmachines.
Thereasonyouwouldwanttosplitoutandruntheseparateserviceswould
reducedowntothefollowingobservations:
1. You have completely autonomous teams (even geographically
separated)thatdon’tknowabouttheotherservicesteamsandeachis
creatinganddeployingservicesindependently.
2. Eachservicegetsdeployedonthenumberofmachinesrequiredto
handletheloaditneedstosupport.
Youwould monitor the transactions and loadingneeds of eachserviceand
deploy them out to your cluster as needed. The point of a sophisticated PaaS
solutionforaMicroservicesarchitectureisthatitwouldhandlethisdistribution
andshufflingofservicestobalancethemacrossmachines.
Microservicesframeworks
If you have split up your Node.js application into several smaller
applications,eachneedingtheirown“npmstart”commandtorun,thenyouwill
wanttoconsiderusingDockerforeachofthesediscreetNode.jsapplications.To
do that, you will want to use something like Amazon ECS, EKS, or Fargate.
ThereisalsoOpenShiftOnline,Kubernetes,DockerSwarm,CloudFoundry,or
NGINXUnit(ApplicationServerandServiceMesh).Theseframeworksmanage
thedeploymentofyourDockercontainerstoaclusterofmachines.
Really sophisticated Microservices architectures may even employ some
typeofservicediscoveryregistry.ThereareframeworkssuchasScenicathatare
availabletodothis.Youneedtojustifythatyoureallyneedthiscomplexityand
overhead.
ThereisaNode.jspackagenamedHydrathatyoucandownloadfromNPM
that might be of benefit to you is you are creating a more complex
implementation. Hydra gives you things like service discovery, distributed
messaging,messageloadbalancing,logging,presence,andhealthmonitoring.
Queuesandworkerprocesses
A common approach in backend services is to take requests that require
longer running processing and accept that request immediately and return
success,butactuallyjustqueueitup.Imagineyouhadaphotostorageservice
that has an HTTP Post endpoint API that accepts images. You might want to
createthumbnailimagesanddosomeimageprocessingtodofacialrecognition
andaddmetadatatoidentifypeople.
Tokeepyourservicescalable,youcouldhaveyourwebserviceacceptthe
image,copyittoS3andthenplaceanentryintoMongoDBorAWSSQStobe
lookedat later. Then youcould have separateEC2 machinesor AWSLambda
functions that scale and independently read the work requests and do the
processing. This is how to manage the work and keep the web service very
responsive.
Chapter12:NewsWatcherApp
Development
Itis now timeto beginconstructinga Node.jsService layer thatintegrates
withtheMongoDBdatalayerfrompartoneofthisbook.Thischaptertakesthe
concepts you have already learned regarding Node and applies them in a real
project.WhatyouwillbecreatingisaRESTfulwebAPIthatyourpresentation
layerwillbeabletointegratewith.Youwilllearnhowtoimplementeverything
needed for a fully functional cloud-hosted web service. You will utilize best
practicesfortestingandDevOpsinthechaptersthatfollow.
Note:Don’tforgetthatyoucanaccessallofthecodefortheNewsWatcher
sampleprojectathttps://github.com/eljamaki01/NewsWatcher2RWeb.
12.1InstalltheNecessaryTools
OneoftheamazingthingsaboutNodeishowsimpleandquickitistosetup
aserver.ItcanbehostedinAWSasaPaaSofferingusingElasticBeanstalk,or
otherhostinginfrastructure,orevenbedeployedandmanagedmanually.Node.js
runsonmanydifferentoperatingsystems.
Togetstarted,installthefollowing:
AcodeeditorsuchasVisualStudioCode,Sublimetext,Vim,etc.
Node.jsfromhttps://nodejs.org/.Getastableversion.Thisinstallsthe
nodeexecutableforyouandalsoinstallstheNPMexecutable.
Note: Visual Studio Code is not the same as Visual Studio. VS Code is a
completelynewtoolthatoffersaricheditingenvironmentaswellasintegrated
featuresforsourcecodecontrolanddebugging.VSCodehasthecapabilityto
launchtasksthroughthemeansoftoolslikeGulpwithnoneedtojumpouttoa
commandline.Thesetoolscanbeusedtoautomatebuildandteststepsthatyou
need to run frequently. With Visual Studio Code, you will be able to create a
project and run it locally on your machine and have access to IntelliSense,
debugging,andwebapppublishingthroughGit/GitHub.
12.2CreateanExpressApplication
Start by creating a folder for your application. I named mine
“NewsWatcher2RWeb”. You can now create the minimum amount of code
requiredforaNode.jsapplication.Tohelpmaintainyoursanity,youshouldstart
withthesmallestamountofcodepossibleandpushitallthewaytodeployment.
This will eliminate unneeded investigation time of issues unrelated to just
gettingthebasicserviceupandworking.
Launch Visual Studio Code and click File->Open Folder, then select the
“NewsWatcher2RWeb”folderyoujustcreated.WithVSCodeopen,youseethe
EXPLORE view open and in there find two subfolders. You can also use a
commandpromptandtype“code.”toopenupVSCodeintheprojectdirectory.
Figure52-VSCodeUI
TheNewsWatcher2RWebfoldershowsyouallfilesandsubfolders.
Youcan now get started and create a simple Node.js application, and then
deployittoanAWSElasticBeanstalkEC2instance.Oncethatisverifiedand
runningok,youcanaddmorecodetofilloutthefullfunctionalityoftheweb
service.
On the NEWSWATCHER subfolder, click on the icon to create new files.
Startbycreatingthesethreefiles:
.gitignore
package.json
server.js
The.gitignorefilewillnotbeusedrightnow,butwouldcomeintousewhen
you make use of Git and GitHub. You list files and directories you want to
excludefromGitsourcecodecontrol.
Atthis point,there is nonode_modulesfolder yet.It will automaticallybe
createdwhenyouinstallthenodemoduleswiththe“npminstall”command.
Addthefollowingcodetotheserver.jsfile:
varexpress=require('express');
varapp=express();
app.get('/',function(req,res){
console.log('Sendmessageongetrequest');
res.send('Hellofull-stackdevelopment!');
});
app.set('port',process.env.PORT||3000);
varserver=app.listen(app.get('port'),function(){console.log('Expressserverlisteningonport:'+
server.address().port);
});
Addthefollowinglinestothepackage.jsonfile:
{
"name":"NewsWatcher",
"version":"0.0.0",
"description":"NewsWatcher",
"main":"server.js",
"author":{
"name":"yourname",
"email":""
},
"scripts":{
"start":"nodeserver.js"
},
"dependencies":{
"express":"^4.13.4"
}
}
Saveallthefiles,thenopenacommandpromptwindowandnavigatetoyour
project’sdirectory.Atthecommandprompt,typenpminstall.Thiswilllookat
your package.json file, and install the standard Node modules along with the
Express module that is listed as a dependency. A new directory will be added
withthenamenode_modules.
You can now try running your Node project locally. Once it is proven to
function,youwillworkongettingitdeployedtoAWS.Atthecommandprompt,
type"npmstart".Youcanalsotype"nodeserver.js"torunit.Iftheprojectruns
successfully,youwillseethefollowingconsoleoutput:
Figure53-localconsoleoutput
Open a web browser and navigate to http://localhost:3000/. In the browser
window,youwillseeyourmessage:
Figure54-Projectmessageinabrowserwindow
12.3DeployingtoAWSElasticBeanstalk
It is now time to create your Elastic Beanstalk app through the AWS
Management Console. To do so, you must already have an AWS account. Be
awarethatyoumayincursomecostatthispointifyouarenotrunningwitha
freeaccountwithAWS!
Tocreatetheapp:
1. Open a web browser and navigate to
https://console.aws.amazon.com/console/.
2. In the upper right corner of the web page, for Region, select a
region such as US East (N. Virginia) as the region you want your
servicestoberunningin.
Figure55-Regionselection
3. Fromtheselectionofservices,clickElasticBeanstalk:
Figure56-SelectElasticBeanstalk
4. ClickCreate New Application. Enter a name and description. I
gaveitthename“newswatcher”.
Figure57-ElasticBeanstalkcreatenewapplication
5. Click to create a new environment by clicking Createone now.
SelectaWebserverenvironment.
6. SelectNode.jsastheplatformandclickCreateenvironment.
Oncethesiteisready,theelasticbeanstalkportallooksasfollows:
Figure58-ElasticBeanstalkportal
ClicktheURL:linkatthetopofthepagetoseeyoursiteworking.
Figure59-Launchthesite
Now you can deploy your own simple Express application.On a windows
machine,thisisasfollows:
1. UsingtheWindowsFileExplorer,navigatetoyourprojectfolder.
2. Selectthepackage.jsonandserver.jsfilestogether,thenright-click
andselectSendto->Compressed(zipped)folder.Givethezipfilea
nameandsaveit.
3. In the Elastic Beanstalk dashboard for the newswatcher
application,clickUploadandDeploybuttonandselectyourzipfile.
4. Wait for the confirmation that the deployment is ready and click
theURLagain.
YourNode.jsapplicationisnowworkingforeveryonetosee.
Youdonotneedtozipandsendthenode_modulesfolder.Thedeploymentto
ElasticBeanstalkwillrun'npminstall'foryouontheEC2instances.Onething
you should do is to set an environment variable through the Elastic Beanstalk
management console so that the node install actually becomes “npm install
production”. Set the environment variable in the Configuration -> Software
Configuration->EnvironmentProperties.as:
Propertyname:NPM_CONFIG_PRODUCTIONPropertyvalue:true
This will make the install go much faster as it will not deploy any npm
modulesthatareneededintestordevelopmentenvironments.
Figure60-Uploadanddeploythezipfile
Youjustperformedamanualdeployment.Whileafewmanualdeployments
mightbe tolerable, you eventually want full continuous-integrationscripts that
runtestsanddeploymentsforyou.
12.4BasicProjectStructure
YoucannowaddtherestofthecodefortheNewsWatcherapplication.You
canstartby addinginthe restofthe Node.jsdependenciesthat youwillneed.
Edityourpackage.jsonfiletobeasfollows,thensaveit.Itshouldlooksimilarto
thefollowing.YoucanrefertotheGitHubprojectforthecompletefile.
{
"name":"newswatcher",
"version":"0.0.1",
"main":"server.js",
"scripts":{
"start":"nodeserver.js",
},
"dependencies":{
"async":"^2.6.0",
"bcryptjs":"^2.4.3",
"body-parser":"^1.18.2",
"dotenv":"^4.0.0",
"express":"^4.16.3",
"express-rate-limit":"^2.11.0",
"helmet":"^3.12.0",
"helmet-csp":"^2.7.0",
"joi":"^13.3.0",
"jwt-simple":"^0.5.1",
"mongodb":"^3.0.8",
"morgan":"^1.9.0",
"response-time":"^2.3.2"
},
"devDependencies":{
"eslint":"^4.19.1",
"eslint-plugin-react":"^7.8.1",
"mocha":"^4.1.0",
"selenium-webdriver":"^3.6.0",
"supertest":"^3.0.0"
}
}
Nowopenacommandpromptwindowandtypenpminstallatthecommand
prompt.Youwilladdcodetotherestofthefileslaterinthischapter.Youcan,of
course, go to the GitHub project and get all of the code for the NewsWatcher
sampleapplication.Youarenowreadytoputyourservicelayerwebapplication
RESTfulAPItogether.
Note:Ifyoueverwanttocompletelyupdatetothelatestversionofone or
moreinstalledpackagesinyourpackage.json,changeeveryversionnumberto
“*”,andthenrun–“npmupdate-–save”and“npmupdate–save-dev”.
12.5WhereitAllStarts(server.js)
ThestartingpointofyourNodeapplicationcanbeinafilenamedwhatever
youlike,Ipreferserver.js.ThisfileinstructsNodewhattodotobeinitialized
andwhattosubsequentlyexecute,onceupandrunning.Inreality,thisfilecould
benamedanything.ThisfiledoesthingslikeestablishaMongoDBconnection,
set up error handling, and establish the Express listener for the HTTP request
handling.
Atthetopoftheserver.jsfileiswhereyouwillplacetherequirestatements
thatloadtheneededmodules.Ifyourecall,modulereferencesinsideafileare
internaltothatfile.
Someofthemodulesyouwillspecifyasbeingrequiredareactuallyusedas
middlewarefor Express. This means you set them up with a require statement
andthendon’tactuallyusethemdirectly.Theyacttointerceptcallsthroughan
app.use()setting.Ifneeded,youcanreferbacktothechapteronmiddlewareto
reviewthisconcept.
Hereisthefirstsectionofcodeintheserver.jsfile.Ihaveaddedcomments
tobrieflydescribethepurposeofeachrequirestatement.
varexpress=require('express');//Routehandlersandtemplatesusage
varpath=require('path');//Populatingthepathpropertyoftherequest
varlogger=require('morgan');//HTTPrequestlogging
varbodyParser=require('body-parser');//AccesstotheHTTPrequestbody
varcp=require('child_process');//ForkingaseparateNode.jsprocesses
varresponseTime=require('response-time');//Performancelogging
varassert=require('assert');//asserttestingofvalues
varhelmet=require('helmet');//Securitymeasures
varRateLimit=require('express-rate-limit');//IPbasedratelimiter
varcsp=require('helmet-csp');
The first require statement provides the object that will be needed for
leveragingExpress.Thepathmoduleisbasicallyusedtoprovideahelperobject
that will be used to manipulate strings for specifying the file paths in your
project.Therestoftherequirestatementsareforsettingupmodulesthatactas
middleware.
Thesenextlinestoaddwillsetupenvironmentsettingsforthecodethatcan
havethingslikepasswordsandothersecretsandconfigurationsthatareneeded.
Whenrunlocally,thisisreadoutofafilenamed.env.Whenruninproduction,
the file is not used, but the values are set in the AWS Elastic Beanstalk
environmentsettingsandthecodeseesthoseinstead.
if(process.env.NODE_ENV!=='production'){//readinginofsecrets
require('dotenv').config();
}
Thesenextlineswillpullincodefrommodulesthatyouwillwriteyourself
thatprovidetheroutehandlers:
varusers=require('./routes/users');
varsession=require('./routes/session');
varsharedNews=require('./routes/sharedNews');
varhomeNews=require('./routes/homeNews');
Thecodecannowstarttoimplementsomeofitscapabilities.Younowmake
theExpressapplicationobjectavailableandsetasettingonitthatwillbeneeded
forwhenitisruninAWS.SincetheappisbehindanNginxloadbalancerwith
ElasticBeanstalk,youdon’twanttheloadbalancerIPaddresssentintheheader
requests, but want the IP address of the actual machine that it was acting on
behalfof.Thisiswhatthetrustproxysettingdoes.
varapp=express();
app.enable('trustproxy');
Middleware
Next,youcanseehowtheExpressmiddlewareishookedupforsomeofthe
modulesyouareincorporating.Herearethoselines:
//Applylimitstoallrequests
varlimiter=newRateLimit({
windowMs:15*60*1000,//15minutes
max:100,//limiteachIPto100requestsperwindowMs
delayMs:0//disabledelaying-fullspeeduntilthemaxlimit
});
app.use(limiter);
app.use(helmet());//Takethedefaultstostartwith
app.use(csp({
//Specifydirectivesforcontentsources
directives:{
defaultSrc:["'self'"],
scriptSrc:["'self'","'unsafe-inline'",'ajax.googleapis.com',
'maxcdn.bootstrapcdn.com'],
styleSrc:["'self'","'unsafe-inline'",'maxcdn.bootstrapcdn.com'],
fontSrc:["'self'",'maxcdn.bootstrapcdn.com'],
imgSrc:['*']
}
}));
//AddsanX-Response-Timeheadertoresponsestomeasureresponsetimes
app.use(responseTime());
//logsallHTTPrequests.The"dev"optiongivesitaspecificstyling
app.use(logger('dev'));
//Setsuptheresponseobjectinroutestocontainabodypropertywithan
//objectofwhatisparsedfromaJSONbodyrequestpayload
//Thereisnoneedforallowingahugebody,itmightbesomeattack,
//sousethelimitoption
app.use(bodyParser.json({limit:'100kb'}));
//MainHTMLpagetobereturnedisinthebuilddirectory
app.get('/',function(req,res){
res.sendFile(path.join(__dirname,'build','index.html'));
});
//ServingupofstaticcontentsuchasHTMLfortheReact
//SPA,images,CSSfiles,andJavaScriptfiles
app.use(express.static(path.join(__dirname,'build')));
Thiscodetakesthemodulesbroughtinthroughtherequire()statementsand
insertsthemasmiddlewarebycallingapp.use().
ThefirstpieceofmiddlewareiswhatgivesprotectionagainstDoSattacks.
You can look up the module in GitHub to actually see how it works as
middleware.
ThenextpieceofmiddlewareisHelmet.Icovereditsuseearlier.Helmetisa
securitymitigationmodulethattweakstheHTTPheaders.
Therearefiveotherusesofmiddlewarethataredocumentedinthecodeto
tellyouwhattheydo.Eachisveryusefulandyouwillbenefitfromthemall.
You can see the use of the path module to provide functionality to
manipulate path strings with the join function. The __dirname variable is
providedbyNodesothatyoucanuseittogetthenameofthedirectorythatthe
currentlyexecutingcoderesidesin.Inthisusage,itwouldreturnthedirectoryof
theserver.jsfile.Itwillbethelocalpathifyouarerunningitlocally,orwhatever
it is on the AWS production machine if it is running in the deployed
environment.
ThecodingoftheReactWebSPAReactiscoveredinthethirdpartofthis
book. You can see the code that serves up this static content that is using the
Expressstaticmodule.
ForkingaProcess
The next code in the server.js file is used to fork off a separate Node.js
process and give it a file to execute. This is used to shuttle off any code
processingthatismoreintensiveandthatyoudon’twantrunonyourmainNode
processthread.
varnode2=cp.fork('./worker/app_FORK.js');
Earlier,Iexplainedthatyouneedtobecarefulwithcodeyouwritethatwills
executeonthemainNode.jsV8VM.WithNewsWatcheryouneedtooffloada
fewthingstoaseparateNodeprocess.Theseinvolvecodeforcollectingnews
storiesfrom internet sources intoNewsWatchers masterlist in MongoDB and
doingthefilteredmatchingofstoriesforusers.Iwillshowyouthecodeinthe
forkedprocesslater.Rememberalso, thatIsaiditwouldprobablybebetterto
useAWSLambdaforthistypeofcodeprocessing.
You can see that you use the child_process module to start up the second
process.SinceNodeisportedtomanyplatforms,itwillcallwhateverlow-level
codeisneededtoaccomplishthisfortheOSitisrunningon.Nodemakesuseof
some libraries that do this and these have been ported already. You can pass
messagesbackandforthbetweenprocessesifyoulike.Youwillbedoingthis
laterincode.
Iftheforkedprocessisexperiencingruntimeerrors,itcouldshutitselfdown
andthenthemainprocesscouldbesignaledtostartitupagain.Youaddcodein
server.jstorestarttheforkedprocessasfollows:
node2.on('exit',function(code){
node2=undefined;
node2=cp.fork('./worker/app_FORK.js');
});
TheMongoDBDataLayerConnection
Mostofthecodeintheservicelayerdealswithinteractionswiththebackend
data storage layer. You initialize your MongoDB connection by utilizing the
mongodbmodulethatisanNPMdownload.Youusetheconnect()functionand
then set up the usage of the newswatcher collection. The connect() function
takestheMongoDBconnectionURL.
vardb={};
varMongoClient=require('mongodb').MongoClient;
//UseconnectmethodtoconnecttotheServer
MongoClient.connect(process.env.MONGODB_CONNECT_URL,function(err,client){
assert.equal(null,err);
db.client=client;
db.collection=client.db('newswatcherdb').collection('newswatcher');
});
Yousavetheconnectionasapropertyonanobjectyousetupnameddbto
beusedlaterwithyourExpressroutesthroughmiddlewareinjection.Watchfor
thatcodecomingupsoon.
Thelastthingtonoteabouttheabovecodeisthatyouhaveaconfiguration
file(.env)forkeepingsettingsinthatyouwanttohaveinacentralplace.Some
ofthevaluesinthatfileareonesyouwanttokeepsecret.Don’tpostthatfilefor
anyonetosee.Aswasmentionedalready,youkeepneededconfigurationvalues
there,butthesevaluesarealsosetasenvironmentname/valuepairsinanElastic
Beanstalkenvironment.
SharingObjects
Thenode2anddbvariablesareneededinyourroutingcode.Thedbobject
databaseconnectionwillbeusedforalloftheCRUDoperations,soyouneedto
makethat available. Youexpose these variables through middleware injection.
This means that you have a chance to inject the objects into the request
processingchainbyaddingthemaspropertiesontherequestobject.
Youplacea middleware function right atthe top of the Express chainthat
everyrequestwillhavetopassthroughfirst.Inthereyouattachnewproperties
ontherequestobjectthatisbeingpassedalong.
Asrequired, you callnext() to move the execution along to the rest of the
processingchainfortherequest.Rememberthattheusefunctionappliesacross
allrequests,sothataget,putoranyotherrequestisroutedthroughherefirst.
app.use(function(req,res,next){
req.db=db;
req.node2=node2;
next();
});
ExpressRouteHandlers
Youarealmostdonewiththemainapplicationcodethatsetseverythingup.
ThefollowingarealloftheroutesforyourHTTP/RestAPI.Thisgoesbackto
understandingwhatyourobjectsareandwhatverbseachwillsupport.Youjust
list out each object. Inside each of the supporting modules you will find the
verbsandanysub-objectsoffofthose.Thesearemodulesyouwritethatusethe
ExpressRouterobjectasexplainedinsection9.2.
//RestAPIroutes
app.use('/api/users',users);
app.use('/api/sessions',session);
app.use('/api/sharednews',sharedNews);
app.use('/api/homenews',homeNews);
Next,thereisanerrorhandlingroutethathandlesinvalidURLsthatcomein.
Thisreturnsa404codetosignalthattheresourcewasnotfound.Basically,if
noneoftheroutesmatch,thenthisonewill.Forexample,ifarequestcamein
for /api/blah, it would go here. This code activates an express error handler,
becauseitcallsnext(err).
//catcheverythingelseandforwardtoerrorhandlerasa404toreturn
app.use(function(req,res,next){
varerr=newError('NotFound');
err.status=404;
next(err);
});
Herearetheerrorhandlingroutesforwhenyouhaveanerrorreturnedinthe
code.Yougetherewhennext(err)iscalledinanyroutingcode.Thereisalsoa
handlerthatonlykicksinwhenrunninginyourdevelopmentenvironment.You
wanttodothissothatyoucanaddinwhatthestacktraceis.Thesecondhandler
istheonethatkicksin,intheproductionenvironment.
//developmenterrorhandlerthatwilladdinastacktrace
if(app.get('env')==='development'){
app.use(function(err,req,res,next){
res.status(err.status||500).json({message:err.toString(),
error:err});
console.log(err);
});
}
//productionerrorhandlerwithnostacktracesexposedtousers
app.use(function(err,req,res,next){
res.status(err.status||500).json({message:err.toString(),error:{}});
console.log(err);
});
Thefinallinesinserver.jscontainthenecessarycodethattellsExpresstobe
listeningforHTTPrequests.Inproduction,itpicksuptheportnecessarytorun
inthathostedenvironment.Thelastthreelinesshown,exporttheserverforour
testingframeworktouse.
app.set('port',process.env.PORT||3000);
varserver=app.listen(app.get('port'),function(){
console.log('Expressserverlisteningonport'+server.address().port);
});
server.db=db;
server.node2=node2;
module.exports=server;
IfyoulookatthecodeintheGitHubproject,youwillseethatIalsoadded
routehandlersforstartingandstoppingtheV8profilerandfortakingmemory
snapshots.Thesecanthenbeactivatedbyplacingacalltothatparticularroute,
andalsodeactivatedwhennotneeded.
12.6AMongoDBDocumenttoHoldNews
Stories
YouneedtocreateaMongoDBdocumenttostorethecontentsoftheshared
globalnewsstories.Thisisaone-timecreationdoneinadvancethatcanbedone
throughtheMongoDBInc.Compassapplication.Thisdocumentfunctionsasthe
holder for the master list of current news stories and top stories found on the
homepage. This is then used by all users to do matchingof their news filters
with.Thisway,eachuserdoesnotneedtofetchallthenewsindividually.This
documentwillhaveadistinctivevaluesetforthe_idpropertysoyouknowwhat
itisfor.Becarefultonotaccidentallydeleteit.Thisdocumentlooksasfollows:
{
"_id":"MASTER_STORIES_DO_NOT_DELETE",
"newsStories":[],
"homeNewsStories":[]
}
Icoveredthispreviouslyinpartoneofthisbook,soyoumighthavealready
createdthisdocument.Ifnot,youcancreateitatthistime.Thisdocumentcould
becreatedincode ifyoureally wantedto.Since itisa one-timething,Ihave
chosentouseCompasstocreateit.
12.7ACentralPlaceforConfiguration(.env)
To run your Node application locally, there will need to be certain
environmentvariablessetthatyourcodecanreference.Thesearevaluessuchas
thosetoestablishtheconnectiontotheMongoDBdatabase.Youplacean.env
fileintheprojectandthenuseannpmmoduletoconsumethat.Thistechnique
wasusedintheserver.jsfileasalreadyexplained.
Be aware that I cannot divulge the actual contents of my .env file, as you
wouldthenhaveaccesstoservicesIneedtoprotect.Replacevaluesasneededin
yourown.envfile.
//3.6mongodbdriverandlaterconnectionstring
MONGODB_CONNECT_URL=mongodb+srv://babajee:mypass@cluster0-k5ghj.mongodb.net/test
JWT_SECRET=<yoursecretkey>
NEWYORKTIMES_API_KEY=<yoursecretkey>
GLOBAL_STORIES_ID=MASTER_STORIES_DO_NOT_DELETE
MAX_SHARED_STORIES=30
MAX_COMMENTS=30
MAX_FILTERS=5
MAX_FILTER_STORIES=15
The value for the MongoDB connection URL can be found in the Atlas
portal as was explained previously. You go to the clusters page and click the
CONNECTbuttonandfindthestringforconnectingtoanapplication.Follow
theinstructionsthatarefoundthere.Itwilltellyoutoreplacetheplaceholders
foryourownpassword.
12.8HTTP/RestWebServiceAPI
ItisnowtimetofillintheRESTWebServiceAPI.TheAPIwillacceptand
pass back JSON payloads in response to HTTP requests. Eventually, you will
createtheSPAwebpagethatcallsthewebservicebeingdesignedhere.
IfyouthinkabouttheRESTAPIthatyouwanttoexpose,itbecomesclear
that you need to create all the CRUD operations for each resource that is
necessary.Yourresourcesaresessions,users,sharednewsandhomenews.
HereisatablethatlistseverythingtheRESTAPIsupports.Theidparameter
is the identifier of individual resources for a user being accessed. The sid
parameteristheidentifierforstories.
VerbandPath Result
POST
/api/sessions
Createaloginsessiontoken.
DELETE
/api/sessions/:id
Deletealoginsessiontoken.
POST
/api/users
CreateauserwiththepassedinJSONofthe
HTTPbody.
DELETE
/api/users/:id
Deleteasinglespecifieduser.
GET
/api/users/:id
ReturntheJSONofasinglespecifieduser.
PUT
/api/users/:id
Replaceauserwiththepassed-inJSONofthe
HTTPbody.
POST
/api/users/:id/savedstories
Saveastoryforuser,contentofwhichisinthe
JSONbody
DELETE
/api/users/:id/savedstories/:sid
Deleteastorythattheuserhadpreviously
saved.
POST
/api/sharednews
ShareanewsstoryascontainedintheJSON
body
GET
/api/sharednews
Getallofthesharednewsstories
DELETE
/api/sharednews/:sid
Deleteanewsstorythathadbeenshared.
POST
/api/sharednews/:sid/comments
Addacommenttoaspecifiedsharednews
story
GET
/api/homenews
Getallthehomepagenewsstories
You may have noticed that some verbs are missing that you might have
expectedtofind.Forexample,youwillnotseeaGET/api/userstogetthelistof
all users. You don’treally want other people to see everyone that is a user of
NewsWatcher,sodon’tofferthat.Youcertainlycoulddecidetoofferit,butyou
wouldthenwanttoplaceamiddlewarerestrictiononitthatonlyallowslogged
inadministratorstohaveaccesstoit.
An example of a restricted API route that you have is DELETE
/api/users/:id.Ausercanonlydeletethemselves,soyourestrictthattojustthe
logged-inuserfordeletingtheirownaccountandnotanaccountofanyoneelse.
Youcouldwritesomecodetoallowadminstobeabletodeleteanyone.
Rememberthatthetokenisusefulforrestrictingaccesswith.Itisuptoyou
todefinewhatrolesandaccessyouwillneedandthenenforceit.Inthiscase,
each call only works for that account to access that person’s own data for
whateverisauthorized.Perhapsadministratorsthatlogincanbeidentifiedand
allowedaccesstoeverything.
Visualizingthecode
Ifyourecall,thelinesbelowarefoundintheserver.jsfileandareusedtoset
upyourExpressroutehandling.ThefirsttwoExpressapplicationcallsareused
forsendingthefilesbackthattheclientbrowserapplicationwillneed.Thelast
fourExpressapplicationcallsareforroutehandlingofeverythinglistedinthe
RESTAPIresourcetable.
Herearethelinesfromtheserver.jsfileasareminder:
//MainfiletoserveupthatisbuiltbyReactbuildprocess
app.get('/',function(req,res){
res.sendFile(path.join(__dirname,'build','index.html'));
});
//ServingupofstaticcontentsuchasHTMLfortheReactSPA,images,
//CSSfiles,andJavaScriptfiles
app.use(express.static(path.join(__dirname,'build')));
//RestAPIroutes
app.use('/api/users',users);
app.use('/api/sessions',session);
app.use('/api/sharednews',sharedNews);
app.use('/api/homenews',homeNews);
Thefollowingisapictorialrepresentationofhowtheroutingcodeallhooks
together.Thisdoesnotcontainallofthefilesanddetails,butitatleastgivesyou
an idea of the routes that are being serviced. I have even included the UI
rendering React code even though that has not been covered yet. Youcan see
thatIdivideupthecodebythearchitecturallayeritresidesin.
Figure61-NewsWatcherfilediagram
YoucannowlookattheindividualfilesusedforeachoftheExpressroutes
oneatatime.Theyeachexistin theirownfile.Thesearethefileswhereyou
findtheservicingoftheindividualHTTPverbs(GET,POST,PUT,DELETE).
12.9SessionResourceRouting
(routes/session.js)
ThesessionrouteisusedintheAPItoallowpeopletologinandout.Here
arethespecificroutesforthesessionresource:
VerbandPath Result
POST
/api/sessions
Createaloginsessiontoken.
DELETE
/api/sessions/:id
Deletealoginsessiontoken.
Thepostoperationtakesausersemailandpasswordintherequestbodyand
basically logs a user into their account. A token is sent back in the response
body.TheclientcallingcodecantaketheJWTandthenkeeppassingitbackon
subsequentcalls to identify that person. The token canbestored in client-side
storageandusedasneeded.
Thepostverb handler needs to first query for the user document to see if
theyactuallyhavearegisteredaccount.Theusermustexist,ortheycannotbe
logged in. If there is a match for the email, then the stored password hash is
validatedwithahashofthepasswordcomingin.
Ifthepasswordisvalidated,thenatokeniscreatedandpassedbackinthe
JSONpayload.Thetokenscouldbeusedforever,butitwouldbeeasytoadda
timestamp to set it to expire. With that added, you might do something like
requireanewlogineverysixmonths.
Aswasdiscussed,thereisabitofverificationthatcangoonwiththetoken.
Youcheckittomakesureitisoriginatingfromwherethetokenwasoriginally
assignedfrom.TheIPaddressandtheheadersettingforuser-agentarekeptwith
thetokenasadditionalverification.
There is no database storage of the token in the NewsWatcher app, so the
route delete does not really need to do much. There is just a simple check to
verifythatthepersonloggingoutisthesameastheonecontainedinthetoken.
Thecodeusesthejoimodule tovalidatetheincomingrequestbody object
parameters.Hereisthecodeforsession.js:
//session.js:ANode.jsModuleforsessionloginandlogout
"usestrict";
varexpress=require('express');
varbcrypt=require('bcryptjs');//Forpasswordhashcomparing
varjwt=require('jwt-simple');//Fortokenauthentication
varjoi=require('joi');//Fordatavalidation
varauthHelper=require('./authHelper');
varrouter=express.Router();
//
//Createasecuritytokenastheuserlogsinthatcanbepassedtothe
//clientandusedonsubsequentcalls.
//Theuseremailandpasswordaresentinthebodyoftherequest.
//
router.post('/',functionpostSession(req,res,next){
//Passwordmustbe7to15charactersinlengthand
//containatleastonenumericdigitandaspecialcharacter
varschema={
email:joi.string().email().min(7).max(50).required(),
password: joi.string().regex(/^(?=.*[0-9])(?=.*[!@#$%^&*])[a-zA-Z0-9!@#$%^&*]
{7,15}$/).required()
};
joi.validate(req.body,schema,function(err){
if(err)
returnnext(newError('Invalidfield:password7to15(onenumber,onespecialcharacter)'));
req.db.collection.findOne({type:'USER_TYPE',email:req.body.email},
function(err,user){
if(err)
returnnext(err);
if(!user)
returnnext(newError('Userwasnotfound.'));
bcrypt.compare(req.body.password,user.passwordHash,
functioncomparePassword(err,match){
if(match){
try{
vartoken=jwt.encode({authorized:true,sessionIP:req.ip,
sessionUA:req.headers['user-agent'],
userId:user._id.toHexString(),
displayName:user.displayName},
process.env.JWT_SECRET);
res.status(201).json({displayName:user.displayName,
userId:user._id.toHexString(),
token:token,msg:'Authorized'});
}catch(err){
returnnext(err);
}
}else{
returnnext(newError('Wrongpassword'));
}
});
});
});
});
//
//Deletethetokenasauserlogsout
//
router.delete('/:id',authHelper.checkAuth,function(req,res,next){
//Verifythepassedinidisthesameasthatintheauthtoken
if(req.params.id!=req.auth.userId)
returnnext(newError('Invalidrequestforlogout'));
res.status(200).json({msg:'Loggedout'});
});
module.exports=router;
Notice the use of the middleware function authHelper.checkAuth. This is
something you will define next. This is what allows you to inject an
authorizationcheckbeforeproceedingtothefinalfunctionthatdoesthework.If
the authorization fails then the function for the end route handling will not be
called.
12.10AuthorizationTokenModule
(routes/authHelper.js)
Youhaveseenhowauserlogsinandatokenisgenerated.Younowneedto
create some middleware that will be inserted and run for verifying the token
beforeproceedingonaroutethatneedstobesecure.Thiscodewilllookatthe
passed-in token and make sure it is valid. As you know, middleware can be
insertedintoanyrouteandthatiswhatwillbehappeninghere.
Each of the routing modules will make use of the authHelper module to
verifythatavalidtokenisbeingpassedbeforeperforminganyotheraction.This
happensbecauseeachHTTP/Restcallwouldhaveanx-authheadertokenvalue
filledin.Youcoulduseaheadersuchas“Authorization:Bearer<token>”ifyou
wanttomimicstandardslikeOIDC.
YouwillseetheuseofthecheckAuthfunctioninmanyoftheroutehandlers.
Thecodeherewillsimplyverifythatthereisanx-authheaderand,ifthereis,
thendecodeitwiththesecret,thensetthedecodedobjectinarequestproperty
namedauthforfurtherusagebyanythingintheprocessingchain.Ifthetokenis
missing, has been tampered with, or does not contain what it is supposed to
contain,anerrorisreturned.
Whenyoulookatthepasswordhandlingcodeofsession.js,youseewhere
alltheinformationwasplacedinthetoken.Itreallyisuptoyoutodecidewhat
toputinthere.Withthistoken,youcanrepresenttheuserbeingsignedin.Do
not store the user password (or other sensitive data) in the token. Here is the
middlewarethatverifiesthatatokenisvalid:
//
//authHelper.js:ANode.jsModuletoinjectmiddlewarethatvalidatesthe
//requestheaderUsertoken.
//
"usestrict";
varjwt=require('jwt-simple');
//
//Checkforatokeninthecustomheadersettingandverifythatitis
//signedandhasnotbeentamperedwith.
//Ifnoheadertokenispresent,maybetheuser
//TheJWTSimplepackagewillthrowexceptions
//
module.exports.checkAuth=function(req,res,next){
if(req.headers['x-auth']){
try{
req.auth=jwt.decode(req.headers['x-auth'],process.env.JWT_SECRET);
if(req.auth&&req.auth.authorized&&req.auth.userId){
returnnext();
}else{
returnnext(newError('Userisnotloggedin.'));
}
}catch(err){
returnnext(err);
}
}else{
returnnext(newError('Userisnotloggedin.'));
}
};
IfyouprovideaUIcheck-boxfortheusertostayloggedin,thentheycan
checkthatyoucanstorethetokenonthedeviceandnothavetologthemineach
time. If a user wanted to get their token from the returned login request, they
could. If they passed it on to anyone else, their account could possibly be
compromised.Thus,theextratests.ThereisatestfortheIPaddressbeingthe
same. This may not actually work if you have users logging in from different
devicesallthetimewithdifferentIPaddresses.
12.11UserResourceRouting
(routes/users.js)
The user resource represents information for a logged in user. A user
documentretrievedbytheirIDwillcontaininformationliketheirprofilethathas
theirnewsfiltersandalsothenewsstoriesthathavematched.Foragivenuser,
youcanalsomakeRestcallstosaveastoryordeleteasavedstory.Herearethe
specificroutesfortheuserresource:
VerbandPath Result
POST
/api/users
Createauserwiththepassed-inJSONofthe
HTTPbody.
DELETE
/api/users/:id
Deleteasinglespecifieduser.
GET
/api/users/:id
ReturntheJSONofasinglespecifieduser.
PUT
/api/users/:id
Replaceauserwiththepassed-inJSONofthe
HTTPbody.
POST
/api/users/:id/savedstories
Saveastoryforuser,contentofwhichisinthe
JSONbody
DELETE
/api/users/:id/savedstories/:sid
Deleteastorythattheuserhadpreviouslysaved.
Startbylookingattherequirestatementsattheverytopofusers.js.Someof
themodulesrequiredareonesthatyouhaveseenbefore.
varexpress=require('express');
varbcrypt=require('bcryptjs');
varasync=require('async');
varjoi=require('joi');//Fordatavalidation
varauthHelper=require('./authHelper');
varObjectId=require('mongodb').ObjectID;
varrouter=express.Router();
...codecutouthere...
module.exports=router;
Let’slookattheverbhandlingfunctionsonebyone.
POST/api/users
The post verb takes a JSON payload and creates a new user account as a
documentinyourcollection.Thishappenswhenauseraccountisfirstcreated.
ApasswordispassedinaspartoftheJSONbodyandyoucreateahashofitto
storeintheMongoDBdatabasedocumentforthatuser.
This call will fail if there already exists a document in the collection with
thatemailvaluealready.Relyingonanemailaddresstoidentifyauseraccount
is one way to keep user accounts unique and identifiable. The mongodb
findOne()functioniswhatisusedtoseeifauseraccountexistedalready.
The code goes ahead and creates all of the properties ever needed in the
document.Defaultvaluesareusedthatmakesense.Thereisevenasamplenews
filtersetupfortheuser.
Noticethecalltonode2.send().Thisisusedtopassamessagetotheforked
Nodeprocesstooffloadthefilterprocessingforthatnewlyaddeduser.Thiscall
willtakethenewaccountanddotheinitialmatchingofstoriesforthefilter.I
havenotshownyouthecodefortheforkedprocessyet,butyoucanseethatthe
conceptisverysimple.
Whathappensrightatthestart,isthevalidationofthepassed-inJSONbody.
Youwant tomakesure thatthereare noextraproperties, andvalidatethat the
allowedonesconformtosomeknowntypesandhavesafevalues.ThejoiNPM
module is used to perform the validations. Here is the code for the post verb
handler:
router.post('/',functionpostUser(req,res,next){
//Passwordmustbe7to15charactersinlengthandcontainatleastone
//numericdigitandaspecialcharacter
varschema={
displayName:joi.string().alphanum().min(3).max(50).required(),
email:joi.string().email().min(7).max(50).required(),
password: joi.string().regex(/^(?=.*[0-9])(?=.*[!@#$%^&*])[a-zA-Z0-9!@#$%^&*]
{7,15}$/).required()
};
joi.validate(req.body,schema,function(err,value){
if(err)
returnnext(newError('Invalidfield:displayname3to50alphanumeric,validemail,password7to
15(onenumber,onespecialcharacter)'));
req.db.collection.findOne({type:'USER_TYPE',email:req.body.email},function(err,doc){
if(err)
returnnext(err);
if(doc)
returnnext(newError('Emailaccountalreadyregistered'));
varxferUser={
type:'USER_TYPE',
displayName:req.body.displayName,
email:req.body.email,
passwordHash:null,
date:Date.now(),
completed:false,
settings:{
requireWIFI:true,
enableAlerts:false
},
newsFilters:[{
name:'TechnologyCompanies',
keyWords:['Apple','Microsoft','IBM','Amazon','Google','Intel'],
enableAlert:false,
alertFrequency:0,
enableAutoDelete:false,
deleteTime:0,
timeOfLastScan:0,
newsStories:[]
}],
savedStories:[]
};
bcrypt.hash(req.body.password,10,functiongetHash(err,hash){
if(err)
returnnext(err);
xferUser.passwordHash=hash;
req.db.collection.insertOne(xferUser,functioncreateUser(err,result){
if(err)
returnnext(err);
req.node2.send({msg:'REFRESH_STORIES',doc:result.ops[0]});
res.status(201).json(result.ops[0]);
});
});
});
});
});
Users are allowed to sign up for an account by having them provide their
username, email, and a password. You will not store the actual password, but
will instead store an encrypted hashed value of the password. Even if anyone
weretogetaholdofthatforauser,theywouldstillnotbeabletologinwithit
asitisextremelydifficulttodecryptthatintothepassword.
Onceapersonisregisteredasauser,thenthe/api/session/pathwillbeused
toaccepttheiremailandpasswordtogettheirsessiongoingeachtimetheywant
tologinanduseNewsWatcher.
Asexplainedinthechapteronauthenticationandauthorization,youwillbe
sendingatokenintheheaderofeachHTTP/Restrequest.Whenauserlogsin,
they get a token that is subsequently used to identify them for all further
interactions.
DELETE/api/users/:id
With this path, you see the use of a passed in id. It comes in through the
mechanismofExpress.SpecifyingthepathlikethiswillhaveExpresscreatea
propertyofreq.params.id.Whatyouneedtodoistoverifythattherequestfora
deletion of a user is actually the id that exists in the token. This way a user
cannotdeleteanaccountthatdoesnotbelongtothem.
Lookatsession.jsagainandyouseewherethemongodb_idpropertyofthe
retrieveddocumentiscaptured. Thisiswhat isgoing tobepassed backinthe
RestrequestURLpathportiontoidentifyauser.
The middleware function authHelper.checkAuth is injected to do the
verification that a valid token exists for the request. That middleware-injected
function will return an error if the token is not acceptable and then the route
functionwillnevergetcalled.
If everything proceeds correctly, the route function executes, and the
document is removed from your collection, and the user account is gone.
findOneAndDelete()isusedastherewouldbeoneandonlyonedocumentwith
that_id.
Thereisahelperfunctionfromthemongodbmoduletotakethestringand
getitintotheproperformneeded.Hereisthecodefortheuserdeletionhandler:
router.delete('/:id',authHelper.checkAuth,function(req,res,next){
//Verifythatthepassedinidtodeleteisthesameasthatinthe
//authtoken
if(req.params.id!=req.auth.userId)
returnnext(newError('Invalidrequestforaccountdeletion'));
//MongoDBshoulddotheworkofqueuingthisupandretryingifthereis
//aconflict,Accordingtotheirdocumentation.
//Thisrequiresawritelockontheirpart.
req.db.collection.findOneAndDelete(
{type:'USER_TYPE',_id:ObjectId(req.auth.userId)},
function(err,result){
if(err){
console.log("POSSIBLEUSERDELETIONCONTENTION?err:",err);
returnnext(err);
}elseif(result.ok!=1){
console.log("POSSIBLEUSERDELETIONERROR?result:",result);
returnnext(newError('Accountdeletionfailure'));
}
res.status(200).json({msg:"UserDeleted"});
});
});
GET/api/users/:id
This route handler retrieves a single user by their id. The app would have
alreadycalledtogetasessiontokenandthenhaveaccesstotheidoftheuserto
passitintothisAPIcalltoretrievetheuserdocument.Sinceyouactuallyhave
the_id(objectid),youcanretrievethedocumentfasterthanifyouhadtoquery
foritsomeotherway.Thereisalwaysanindexcreatedforthe_idproperty.
Theretrievalisdoneandpopulatedwithatransferobject.Noticethatyouare
alsotweakingthe HTTPheaderfor theresponse. Thatisnecessary inorder to
stop caching from happening. Otherwise, when you get to the UI presentation
codeandaretryingtoretrieveauser,youmightnotgetthemostup-to-dateone.
Hereisthecodefortheroutehandlertogetasingleuserbytheirid:
router.get('/:id',authHelper.checkAuth,function(req,res,next){
//Verifythatthepassedinidtodeleteisthesametheauthtoken
if(req.params.id!=req.auth.userId)
returnnext(newError('Invalidrequestforaccountfetch'));
req.db.collection.findOne({type:'USER_TYPE',
_id:ObjectId(req.auth.userId)},
function(err,doc){
if(err)returnnext(err);
varxferProfile={
email:doc.email,
displayName:doc.displayName,
date:doc.date,
settings:doc.settings,
newsFilters:doc.newsFilters,
savedStories:doc.savedStories
};
res.header("Cache-Control","no-cache,no-store,must-revalidate");
res.header("Pragma","no-cache");
res.header("Expires",0);
res.status(200).json(xferProfile);
});
});
PUT/api/users/:id
Aputisusedtoupdateauser,suchasinthecasewheretheyhavealtered
thiernewsfilters.Thecodeisverysimilartowhatyouneededfortheinitialpost
of the user, except you now need to worry about a conflict happening upon a
databasewriteoperation.Hereisthecodefortheuserupdatehandler:
router.put('/:id',authHelper.checkAuth,function(req,res,next){
//Verifythatthepassedinidisthesameasthatintheauthtoken
if(req.params.id!=req.auth.userId)
returnnext(newError('Invalidrequestforaccountdeletion'));
//LimitthenumberofnewsFilters
if(req.body.newsFilters.length>process.env.MAX_FILTERS)
returnnext(newError('ToomanynewsnewsFilters'));
//clearoutleadingandtrailingspaces
for(vari=0;i<req.body.newsFilters.length;i++){
if("keyWords"inreq.body.newsFilters[i]&&
req.body.newsFilters[i].keyWords[0]!=""){
for(varj=0;j<req.body.newsFilters[i].keyWords.length;j++){
req.body.newsFilters[i].keyWords[j]=
req.body.newsFilters[i].keyWords[j].trim();
}
}
}
//ValidatethenewsFilters
varschema={
name:joi.string().min(1).max(30).regex(/^[-_a-zA-Z0-9]+$/).required(),
keyWords:joi.array().max(10).items(joi.string().max(20)).required(),
enableAlert:joi.boolean(),
alertFrequency:joi.number().min(0),
enableAutoDelete:joi.boolean(),
deleteTime:joi.date(),
timeOfLastScan:joi.date(),
newsStories:joi.array(),
keywordsStr:joi.string().min(1).max(100)
};
async.eachSeries(req.body.newsFilters,function(filter,innercallback){
joi.validate(filter,schema,function(err){
innercallback(err);
});
},function(err){
if(err){
returnnext(err);
}else{
//MongoDBimplementsoptimisticconcurrencyforus.
//Wewerenotholdingontothedocumentanyway,sowejustdoa
//quickreadandreplaceofjustthosepropertiesandnotthe
//completedocument.
//Itmattersifnewsstorieswereupdatedinthemeantime(i.e.
//usersattheretakingtheirtimeupdatingtheirnewsprofile)
//becausewewillforcethattoupdateaspartofthisoperation.
//Weneedthe{returnOriginal:false},soatestcouldverifywhat
//happened,otherwisethedefaultistoreturntheoriginal.
req.db.collection.findOneAndUpdate({type:'USER_TYPE',
_id:ObjectId(req.auth.userId)},
{$set:{settings:{requireWIFI:req.body.requireWIFI,
enableAlerts:req.body.enableAlerts},
newsFilters:req.body.newsFilters}},
{returnOriginal:false},
function(err,result){
if(err){
console.log("+++POSSIBLEUSERPUTCONTENTIONERROR?+++err:",
err);
returnnext(err);
}elseif(result.ok!=1){
console.log("+++POSSIBLECONTENTIONERROR?+++result:",
result);
returnnext(newError('UserPUTfailure'));
}
req.node2.send({msg:'REFRESH_STORIES',doc:result.value});
res.status(200).json(result.value);
});
}
});
});
Noticethecodetolimitthenewsfiltersize.YouwillputcodeintheUIto
limit that as well, but that could be hacked in the browser, or by someone
sendingabogusputrequest.You have to guard against potential tampering as
youwouldotherwisehaveacrash,oratleastafailureoftheMongoDBupdate.
Thereistheuseofthis amazinglyusefulmodulecalledasync.This allows
calls to the joi library to happen over and over and allows waiting for each
callbacktoreturnforeachofthefiltersforauser.Whenallareprocessed,the
finalanonymousfunctioniscalled.
The$setoperation is usedto update only individualproperties and notthe
entiredocument.
Thereissomeerrorcheckingcodeintheretodetectanycontentionerror.I
hadtriedoverandoverandhaveneverseenonehappen.
POST/api/users/:id/savedstories
In the user document, there is an array used for saving stories that a user
wants to keep around. This route will take the JSON of the passed-in request
body as the story to save. The id in the route is the id of the user that is
requestingthesavingofthestory.
Thereareafewchecksthatneedtohappenbeforesavingastory.Youneed
toverifythatthestoryisnotalreadyinserted.Thereisalsoalimitonthenumber
ofstoriesthatcanbesaved,sothathastobecheckedalso.
Storieshaveanidassociatedwiththemtobeabletoidentifythemincases
likethiswhereyoudon’twantduplicatessavedorshared.Youwilllaterseethe
codethatcreatesthatid.Hereisthecodeforpostingastorytobesaved:
router.post('/:id/savedstories',authHelper.checkAuth,function(req,res,
next){
//Verifythattheidtodeleteisthesameasintheauthtoken
if(req.params.id!=req.auth.userId)
returnnext(newError('Invalidrequestforsavingstory'));
//Validatethebody
varschema={
contentSnippet:joi.string().max(200).required(),
date:joi.date().required(),
hours:joi.string().max(20),
imageUrl:joi.string().max(300).required(),
keep:joi.boolean().required(),
link:joi.string().max(300).required(),
source:joi.string().max(50).required(),
storyID:joi.string().max(100).required(),
title:joi.string().max(200).required()
};
joi.validate(req.body,schema,function(err){
if(err)
returnnext(err);
//makesure:
//A.Storyisnotalreadyinthere.
//B.Welimitthenumberofsavedstoriesto30
//Notallowedatfreetier!!!
//req.db.collection.findOneAndUpdate({type:'USER_TYPE',
//_id:ObjectId(req.auth.userId),
//$where:'this.savedStories.length<29'},
req.db.collection.findOneAndUpdate({type:'USER_TYPE',
_id:ObjectId(req.auth.userId)},
{$addToSet:{savedStories:req.body}},
{returnOriginal:true},
function(err,result){
if(result&&result.value==null){
returnnext(newError('Overthesavelimit,
orstoryalreadysaved'));
}elseif(err){
console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);
returnnext(err);
}elseif(result.ok!=1){
console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);
returnnext(newError('Storysavefailure'));
}
res.status(200).json(result.value);
});
});
});
Note:The$whereabovedoesnotworkwiththefreetierofAtlasandmustbe
omitted.
DELETE/api/users/:id/savedstories/:sid
Thisissimilartotheotherfunctionsandaccomplishestheverificationofthe
storyexisting,beforebeingabletodeleteit.The$pulloperatorisusedwiththe
arraypropertyofthedocumenttodeletetheentry.Hereisthecodefordeletinga
savedstory:
router.delete('/:id/savedstories/:sid',authHelper.checkAuth,function(req,res,next){
if(req.params.id!=req.auth.userId)
returnnext(newError('Invalidrequestfordeletionofsavedstory'));
req.db.collection.findOneAndUpdate({type:'USER_TYPE',
_id:ObjectId(req.auth.userId)},
{$pull:{savedStories:{storyID:req.params.sid}}},
{returnOriginal:true},
function(err,result){
if(err){
console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);
returnnext(err);
}elseif(result.ok!=1){
console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);
returnnext(newError('Storydeletefailure'));
}
res.status(200).json(result.value);
});
});
12.12HomeNewsRouting
(routes/homeNews.js)
HomenewsstoriesarethosestoriesthatarevisiblewhentheapplicationUI
isfirstseenbytheuser.ThereisonlyonesinglerouteneededforthehomeNews
resource.Auserdoesnotneedtobeloggedintoseethesestories.
VerbandPath Result
GET
/api/homenews
Getallofthetopnewsstories
Atthetopandbottomofthefileistheusualcodetoexposethemoduleas
shownhere:
"usestrict";
varexpress=require('express');
varrouter=express.Router();
...thispartleftout...
module.exports=router;
Hereistheroutehandlerforsettingthenewsstoriesforthehomepage.
GET/api/homenews
Retrievingalltopnewsstoriesisdonebydirectlygettingthearraythatholds
themfromouronesingledocumentforthatpurpose.Theyarethesameforall
users.
router.get('/',function(req,res,next){
req.db.collection.findOne({ _id: process.env.GLOBAL_STORIES_ID }, { homeNewsStories: 1 },
function(err,doc){
if(err)
returnnext(err);
res.status(200).json(doc.homeNewsStories);
});
});
The array also works great, as you can send that back in the response and
thenReactcanbinditontheclientside.
12.13SharedNewsRouting
(routes/sharedNews.js)
Shared news stories are those that are seen by all users. People can save,
view and comment on news stories. Here are the specific routes for the
sharedNewsresource:
VerbandPath Result
POST
/api/sharednews
ShareanewsstoryascontainedintheJSON
body
GET
/api/sharednews
Getallofthesharesnewsstories
DELETE
/api/sharednews/:sid
Deleteanewsstorythathasbeenshared.
POST
/api/sharednews/:sid/comments
Addacommenttoaspecifiedsharednews
story
Atthetopandbottomofthefileistheusualcodeasshownhere:
"usestrict";
varexpress=require('express');
varjoi=require('joi');//Fordatavalidation
varauthHelper=require('./authHelper');
varrouter=express.Router();
...thispartleftout...
module.exports=router;
Hereareeachoftheroutepathhandlers.
POST/api/sharednews
Thiscodeisverysimilartothecodeyoualreadysawforsavingastoryin
user.js.Theonlydifferenceisthatnow,thestoryisbeingcopiedintoadifferent
documentwhereallNewsWatcheruserscanviewstoriesandcommentonthem.
Thereisalimitsetforthenumberofpossiblesharedstories.Thereisalsoa
test to make sure the story was not already shared. If all looks good, the
documentiscreated.Hereisthecodeforsharingastory:
router.post('/',authHelper.checkAuth,function(req,res,next){
//Validatethebody
varschema={
contentSnippet:joi.string().max(200).required(),
date:joi.date().required(),
hours:joi.string().max(20),
imageUrl:joi.string().max(300).required(),
keep:joi.boolean().required(),
link:joi.string().max(300).required(),
source:joi.string().max(50).required(),
storyID:joi.string().max(100).required(),
title:joi.string().max(200).required()
};
joi.validate(req.body,schema,function(err){
if(err)returnnext(err);
//Wefirstmakesurewearenotatthecountlimit.
req.db.collection.count({type:'SHAREDSTORY_TYPE'},
function(err,count)
{
if(err)returnnext(err);
if(count>process.env.MAX_SHARED_STORIES)
returnnext(newError('Sharedstorylimitreached'));
//Makesurethestorywasnotalreadyshared
req.db.collection.count({type:'SHAREDSTORY_TYPE',
_id:req.body.storyID},function(err,count)
{
if(err)returnnext(err);
if(count>0)
returnnext(newError('Storywasalreadyshared.'));
//Wesettheidandguaranteeuniquenessorfailurehappens
varxferStory={
_id:req.body.storyID,
type:'SHAREDSTORY_TYPE',
story:req.body,
comments:[{
displayName:req.auth.displayName,
userId:req.auth.userId,
dateTime:Date.now(),
comment:req.auth.displayName+
"thoughteveryonemightenjoythis!"
}]
};
req.db.collection.insertOne(xferStory,
functioncreateUser(err,result)
{
if(err)
returnnext(err);
res.status(201).json(result.ops[0]);
});
});
});
});
});
GET/api/sharednews
Retrievingallsharedstoriesisdonebydirectlygettingthedocumentsoftype
SHAREDSTORY_TYPEasfollows:
router.get('/',authHelper.checkAuth,function(req,res,next){
req.db.collection.find({type:'SHAREDSTORY_TYPE'}).toArray(function(err,docs){
if(err)
returnnext(err);
res.status(200).json(docs);
});
});
Youknowtherewillnotbemorethan30soitisoktohaveanarrayreturned
andnotuseacursortoiteratethroughtheresults.Thearraytypeworksgreat,as
youcansendthatbackintheresponseandthenReactcanbinditontheclient
side.
DELETE/api/sharednews/:sid
Individualsharedstoriescanbedeleted.Youwillnotactuallybecallingthis
fromthepresentationlayer,butneeditjustfortestingpurposestocleanupafter
yourself.Itcaneitherbecommentedoutorhavesomechecksdonetoonlyallow
anadminaccounttocallit.Thecodeisasfollows:
router.delete('/:sid',authHelper.checkAuth,function(req,res,next){
req.db.collection.findOneAndDelete({type:'SHAREDSTORY_TYPE',
_id:req.params.sid},function(err,result)
{
if(err){
console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);
returnnext(err);
}elseif(result.ok!=1){
console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);
returnnext(newError('Sharedstorydeletionfailure'));
}
res.status(200).json({msg:"SharedstoryDeleted"});
});
});
POST/api/sharednews/:sid/comments
Toaddacomment,youneedtheidofthestoryandtheJSONbodyforthe
comment. Since you have a partially normalized design here with separate
documentsforeachstory,youwillnothaveasmuchwritecontentionthatway.
Therewillstillbeconcurrentaccessissuesforeachindividualstoryasmultiple
commentadditionswillpossiblyconflict.ThisiswhythefindOneAndUpdate()
callisused,asitwillhandlethisforyou.
Noticethatthiscanfailifthere arealready30commentsadded.Thereare
threedifferentpartstothequerycriteriaused.Thefirsttwonarrowitdownto
exactly what is being searched for. Then the $where operator is used and the
actualJavaScriptobjectisaccessedtocheckthearraylength.Thesharedstory
commentisaddedasfollows:
router.post('/:sid/Comments',authHelper.checkAuth,function(req,res,next){
//Validatethebody
varschema={
comment:joi.string().max(250).required()
};
joi.validate(req.body,schema,function(err){
if(err)
returnnext(err);
varxferComment={
displayName:req.auth.displayName,
userId:req.auth.userId,
dateTime:Date.now(),
comment:req.body.comment.substring(0,250)
};
//Notallowedatfreetier!!!req.db.collection.findOneAndUpdate({
//type:'SHAREDSTORY_TYPE',_id:req.params.sid,$where:
//'this.comments.length<29'},
req.db.collection.findOneAndUpdate({type:'SHAREDSTORY_TYPE',
_id:req.params.sid},
{$push:{comments:xferComment}},
function(err,result){
if(result&&result.value==null){
returnnext(newError('Commentlimitreached'));
}elseif(err){
console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);
returnnext(err);
}elseif(result.ok!=1){
console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);
returnnext(newError('Commentsavefailure'));
}
res.status(201).json({msg:"Commentadded"});
});
});
});
Note:The$whereabovedoesnotworkwiththefreetierofAtlasandmustbe
omitted.
12.14ForkedNodeProcess
(app_FORK.js)
Younever want to have any compute-intensive code running in your main
Node.js process. If you do, it will overwhelm the V8 JavaScript processing
thread and your web service will become unresponsive. There are reasonable
solutionstothis,suchasforkingoffotherprocessesfromyourmainprocessand
having code execute there. Remember also, that I said it would probably be
bettertouseAWSLambdaforthistypeofprocessing.
In order to architect your application correctly, you need to consider what
needstobemovedofftothesecondaryprocesses.InthecaseofNewsWatcher,
youcanidentifyafewpiecesofcodethatreallyneedtobesentofftoberunon
asecondNode.jsprocessthatiswaitingtodoanyprocessing.
Youcancreateafilenamedapp_FORK.jsandputafewfunctionsofcodein
there.Onesectionofcodewouldbethatwhichisperiodicallyrunonatimerfor
any batch type of work. For example, you need to populate the master news
document with the latest news stories every once in a while and then run
somethingtomatchallofthefiltersoftheusers.
Other code in app_Fork.js signaled to run by sending a message to this
secondNodeprocessfromthemainprocess.Forexample,ifausereveralters
theirfilters,coderunstoupdatethestoriesthatmatchforthatsingleuser.
Let’sstartwiththetopoftheapp_FORK.jsfileandlookattheinitialization
code. The first lines will set up what is required for your module usage. One
thingtonoteisthatthedatabaseconnectioncannotbesharedacrossprocesses,
soyouneedtoestablishthatagainhere.
"usestrict";
varbcrypt=require('bcryptjs');
varhttps=require("https");
varasync=require('async');
varassert=require('assert');
varObjectId=require('mongodb').ObjectID;
varMongoClient=require('mongodb').MongoClient;
varglobalNewsDoc;
constNEWYORKTIMES_CATEGORIES=["home","world","national","business","technology"];
//
//MongoDBdatabaseconnectioninitialization
//
vardb={};
MongoClient.connect(process.env.MONGODB_CONNECT_URL,function(err,client){
assert.equal(null,err);
db.client=client;
db.collection=client.db('newswatcherdb').collection('newswatcher');
console.log("ForkisconnectedtoMongoDBserver");
});
Let’slookathowyoucommunicatebackandforthbetweenNodeprocesses.
ThereisaglobalvariablemadeavailableinNodenamedprocess.Itisusedfor
accessing process-related properties and functions. One of those functions is
send().Itcanbeusedtosendmessagesbacktotheparentprocessthatforkedthe
process.Theon()functionisforhandlingmessagessenttothisforkedprocess
fromthemainprocess.
process.on('message',function(m){
if(m.msg){
if(m.msg=='REFRESH_STORIES'){
setImmediate(function(doc){
refreshStoriesMSG(doc,null);
},m.doc);
}
}else{
console.log('Messagefrommaster:',m);
}
});
Theonemessagesentfromthemainprocessistohandlechangestoausers
filter.Youschedulethehandlingofthatandreturnimmediatelyfromtheevent.
It needs to be delayed, or queued up as you could have other scheduled timer
functions firing off, as well as other message requests coming from different
users. The setImmediate()function is used to set up callbacks to run after I/O
handlinginthissecondnodeprocess.
Refreshofauser’sfilters
Youcannowlookatthefunctionthatreactstoauserwhohasjustupdated
their filters. This would be the code that is run in response to your message
handling.Thebasicalgorithmloopsthrougheachfilterofauser.Foreachfilter,
the code checks if there are any stories in the master news list that match the
keywords.Thereisalimittothenumberofmatchingstories.
Whentheupdateoftheuserdocumenthappens,the$setoperatorisusedand
onlya singleproperty of thedocument isupdated. Just thearray propertythat
holdsthenewsfilters.
Thereiscodeinherefortestingpurposes.Thisisusedtobeabletoverify
youhaveaknownnewsstorytotestforaspecialpredeterminedkeywordstring.
Youwilllaterseehowthisisusedwhenthetestcodeisdiscussed.Hereisthe
codeforuserstorymatching:
functionrefreshStoriesMSG(doc,callback){
if(!globalNewsDoc){
db.collection.findOne({_id:process.env.GLOBAL_STORIES_ID},
function(err,gDoc)
{
if(err){
console.log('FORK_ERROR:readDocument()readerr:'+err);
if(callback)
returncallback(err);
else
return;
}else{
globalNewsDoc=gDoc;
refreshStories(doc,callback);
}
});
}else{
refreshStories(doc,callback);
}
}
functionrefreshStories(doc,callback){
//LoopthroughallnewsFiltersandseekmatchesforallreturnedstories
for(varfilterIdx=0;filterIdx<doc.newsFilters.length;filterIdx++)
{
doc.newsFilters[filterIdx].newsStories=[];
for(vari=0;i<globalNewsDoc.newsStories.length;i++){
globalNewsDoc.newsStories[i].keep=false;
}
//IftherearekeyWords,thenfilterbythem
if("keyWords"indoc.newsFilters[filterIdx]&&
doc.newsFilters[filterIdx].keyWords[0]!=""){
varstoriesMatched=0;
for(vari=0;i<doc.newsFilters[filterIdx].keyWords.length;i++){
for(varj=0;j<globalNewsDoc.newsStories.length;j++){
if(globalNewsDoc.newsStories[j].keep==false){
vars1=globalNewsDoc.newsStories[j].title.toLowerCase();
vars2=
globalNewsDoc.newsStories[j].contentSnippet.toLowerCase();
varkeyword=
doc.newsFilters[filterIdx].keyWords[i].toLowerCase();
if(s1.indexOf(keyword)>=0||s2.indexOf(keyword)>=0){
globalNewsDoc.newsStories[j].keep=true;
storiesMatched++;
}
}
if(storiesMatched==process.env.MAX_FILTER_STORIES)
break;
}
if(storiesMatched==process.env.MAX_FILTER_STORIES)
break;
}
for(vark=0;k<globalNewsDoc.newsStories.length;k++){
if(globalNewsDoc.newsStories[k].keep==true){
doc.newsFilters[filterIdx].newsStories.
push(globalNewsDoc.newsStories[k]);
}
}
}
}
//Forthetestruns,wecaninjectnewsstoriesunderourcontrol.
if(doc.newsFilters.length==1&&
doc.newsFilters[0].keyWords.length==1
&&doc.newsFilters[0].keyWords[0]=="testingKeyword"){
for(vari=0;i<5;i++){
doc.newsFilters[0].newsStories.push(globalNewsDoc.newsStories[0]);
doc.newsFilters[0].newsStories[0].title="testingKeywordtitle"+i;
}
}
//Dothereplacementofthenewsstories
db.collection.findOneAndUpdate({_id:ObjectId(doc._id)},{$set:{"newsFilters":doc.newsFilters}
},function(err,result){
if(err){
console.log('FORK_ERRORReplaceofnewsStoriesfailed:',err);
}elseif(result.ok!=1){
console.log('FORK_ERRORReplaceofnewsStoriesfailed:',result);
}else{
if(doc.newsFilters.length>0){
console.log({ msg: 'MASTERNEWS_UPDATE first filter news length = ' +
doc.newsFilters[0].newsStories.length});
}else{
console.log({msg:'MASTERNEWS_UPDATEnonewsFilters'});
}
}
if(callback)
returncallback(err);
});
}
Thissecondfunctionisusedinmultipleplaces,soIpulleditouttobeonits
own.
Timereventtopopulatethemasternewslist
Everyfewhours,afunctionrunsthatfetchesallnewsstoriesfromthesource
newsserviceprovider.Thismeansthatitworksonabatchofdataandcouldtake
sometimebeforeitfinishes.ThisfunctionwasplacedinNodeprocessbecause
itislong-running.Itisstillsignificantenoughthatitisbettertoplaceitthereso
astonotupsetthemainNodeprocessCPUusage.Thiscouldbeoffloadedtoa
completelydifferentmachinethroughsomeotherqueuingmechanism.
Thefirstthingthathappenswhenthetimeintervalfires,isthesendingifan
HTTP request to the news feed API provided by the New York Times API
service. The results from that are placed into formatted news elements in the
master document newsStories array. Notice the use of the async module to be
abletoloopanumberoftimesandalsoseta.5seconddelaybetweeneachbatch
newsrequestfromNYT.Thisisbecausethereisarestrictionwiththeusagesuch
thatyoucan’tcallitmorethanfivetimesasecond,ortheymaydisableyourIP
addressfromaccessingtheirAPI.
Thereisanidneededforeachstory.AGUIDcouldhavebeengenerated,but
theproblemisthatthesamestorymightappearagaininthenextbatchofnews
andthatwouldcauseproblemsifyouthoughtitwasanewnewsstory.Ahashof
the link willturn out to be the best way to uniquely identify a story. The link
itselfwouldbeunusablelaterinaURLtopassinasanID,butthehashvalue
works,aslongasyoureplacecertaincharactersofit.
Onceallofthenewsstoriesareinplace,youcangothroughalloftheuser
documents and, for each user, do the story matching against the news filters.
This is done in a somewhat tricky way. The async.doWhilst() functionality is
used.Thisway,itcanhandlethedifficultyofmanagingmultipleasynccalls,one
at a time, in a simple way. The code will keep running as long as there is
processingtodo.
Youhave to consider the throughput capability on the MongoDB side and
notoverwhelm it,so itis goodto have thisprocessing serialized.Youneed to
keepplentyofheadroomforyournormaluserinteractions.Hereisthecode:
varcount=0;
newsPullBackgroundTimer=setInterval(function(){
//TheNewYorkTimesnewsservicecan’tbecalledcalledmorethanfive
//timesasecond.Wecallitoverandoveragain,becausethereare
//multiplenewscategoryis,sospaceeachoutbyhalfasecond
vardate=newDate();
console.log("app_FORK:datetimetick:"+date.toUTCString());
async.timesSeries(NEWYORKTIMES_CATEGORIES.length,function(n,next){
setTimeout(function(){
console.log('GetnewsstoriesfromNYT.Pass#',n);
try{
https.get({
host:'api.nytimes.com',
path:'/svc/topstories/v2/'+NEWYORKTIMES_CATEGORIES[n]+
'.json',
headers:{'api-key':process.env.NEWYORKTIMES_API_KEY}
},function(res){
varbody='';
res.on('data',function(d){
body+=d;
});
res.on('end',function(){
next(null,body);
});
}).on('error',function(err){
//handleerrorswiththerequestitself
console.log({msg:'FORK_ERROR',Error:err.message});
return;
});
}
catch(err){
count++;
if(count==3){
console.log('app_FORK.js:shutingdowntimer:'+err);
clearInterval(newsPullBackgroundTimer);
clearInterval(staleStoryDeleteBackgroundTimer);
process.disconnect();
}
else{
console.log('app_FORK.jserror.err:'+err);
}
}
},500);
},function(err,results){
if(err){
console.log('failure');
}else{
console.log('success');
//Dothereplacementofthenewsstoriesinthesinglemasterdoc
db.collection.findOne({_id:process.env.GLOBAL_STORIES_ID},
function(err,gDoc){
if(err){
console.log({msg:'FORK_ERROR',Error:'Errorwiththeglobalnewsdocreadrequest:'+
JSON.stringify(err.body,null,4)});
}else{
gDoc.newsStories=[];
gDoc.homeNewsStories=[];
varallNews=[];
for(vari=0;i<results.length;i++){
try{
varnews=JSON.parse(results[i]);
}catch(e){
console.error(e);
return;
}
for(varj=0;j<news.results.length;j++){
varxferNewsStory={
link:news.results[j].url,
title:news.results[j].title,
contentSnippet:news.results[j].abstract,
source:news.results[j].section,
date:newDate(news.results[j].updated_date).getTime()
};
//Onlytakestorieswithimages
if(news.results[j].multimedia.length>0){
xferNewsStory.imageUrl=news.results[j].multimedia[0].url;
allNews.push(xferNewsStory);
//Populatethehomepagestories
if(i==0){
gDoc.homeNewsStories.push(xferNewsStory);
}
}
}
}
async.eachSeries(allNews,function(story,innercallback){
bcrypt.hash(story.link,10,functiongetHash(err,hash){
if(err)
innercallback(err);
//Onlyaddthestoryifitisnotintherealready.
//StoriesonNYTcanbesharedbetweencategories
story.storyID=hash.replace(/\+/g,'-').
replace(/\//g,'_').replace(/=+$/,'');
if(gDoc.newsStories.findIndex(function(o){
if(o.storyID==story.storyID||o.title==story.title)
returntrue;
else
returnfalse;
})==-1){
gDoc.newsStories.push(story);
}
innercallback();
});
},function(err){
if(err){
console.log('failureonstoryidcreation');
}else{
console.log('storyidcreationsuccess');
globalNewsDoc=gDoc;
setImmediate(function(){
refreshAllUserStories();
});
}
});
}
});
}
});
},240*60*1000);
functionrefreshAllUserStories(){
db.collection.findOneAndUpdate({_id:globalNewsDoc._id},
{$set:{newsStories:globalNewsDoc.newsStories,
homeNewsStories:globalNewsDoc.homeNewsStories}},
function(err,result)
{
if(err){
console.log('FORK_ERRORReplaceofglobalnewsStoriesfailed:',err);
}elseif(result.ok!=1){
console.log('ReplaceofglobalnewsStoriesfailed:',result);
}else{
//ForeachNewsWatcheruser,donewsmatchingontheirnewsFilters
varcursor=db.collection.find({type:'USER_TYPE'});
varkeepProcessing=true;
async.doWhilst(
function(callback){
cursor.next(function(err,doc){
if(doc){
refreshStories(doc,function(err){
callback(null);
});
}else{
keepProcessing=false;
callback(null);
}
});
},
function(){returnkeepProcessing;},
function(err){
console.log('Timer:Refreshedandmatched.err:',err);
});
}
});
}
Note:Anytimeyouusetheseasync functions,youmustbereallycarefulof
howtheyoperatewiththeirasyncandsynccapabilitiesandmakesureyoucall
therequiredcallbackscorrectlyintherightplace.Errorhandlingcanalsobea
bittrickyhere.
Deletingoldstories
Thereneedstobecodethatwilldeletesharedstoriesafteracertainamount
of time. This becomes another timer that goes off periodically to do this
processing.Thecodeisasfollows:
staleStoryDeleteBackgroundTimer=setInterval(function(){
db.collection.find({type:'SHAREDSTORY_TYPE'}).toArray(
function(err,docs){
if(err){
console.log('Forkcouldnotgetsharedstories.err:',err);
return;
}
async.eachSeries(docs,function(story,innercallback){
//Gooffthedateofthetimethestorywasshared
vard1=story.comments[0].dateTime;
vard2=Date.now();
vardiff=Math.floor((d2-d1)/3600000);
if(diff>72){
db.collection.findOneAndDelete({type:'SHAREDSTORY_TYPE',
_id:story._id},
function(err,result){
innercallback(err);
});
}else{
innercallback();
}
},function(err){
if(err){
console.log('stalestorydeletionfailure');
}else{
console.log('stalestorydeletionsuccess');
}
});
});
},24*60*60*1000);
12.15SecuringwithHTTPS
Atthispoint,Ineedtodiscussanimportantsecuritymeasurethatneedstobe
putintoplace.Youjustcan’thostyourRESTAPIendpointwithoutencrypting
trafficbackandforth.YoufixthisbyonlypermittingHTTPSconnections.That
wayalltrafficisencryptedandsignedsoastobetamper-resistantandharderto
eavesdropon.
SincetheNode.jsserviceisexposedthroughtheElasticBeanstalkapp,you
don’tneedtomakeanychangestoyourNode.jscodetosecureit,itisalldone
throughAWS.IfyourNode.jsinstancewasexposeddirectlytotheInternetand
servingupthetrafficdirectly,thenyouwoulddosomesimpleconfigurationon
the Node.js side to install a certificate and key files and then make a few
modificationstotheNodecode.
Inthiscase,ElasticBeanstalkkeepsNodefrombeingdirectlyexposed.The
ElasticBeanstalkserviceactsasareverseproxyandthatiswhereyouneedto
setupSSL.Youwillneedtogetyourowndomainnameandinstallacertificate
foryourElasticBeanstalkservice.
Note:WhenyoulaunchyourapplicationthroughVSCode,itwillnotaccept
HTTPS locally on your machine. However, tests run against a production
deployment must be altered to use HTTPS. Your test code needs to make the
appropriatechangestotheURLbeingtested.
SecuringcommunicationstoMongoDB
Forperformancereasons,youwillwanttoplaceyourdatabaseinthesame
AWS datacenter as your Elastic Beanstalk Node app. As an added bonus of
doingthis,theabilitytosecurethecommunicationbetweenyourNode.jsservice
and your database becomes easier. This is because everything is sent over the
internal data center network and never gets out over the public Internet. As a
furthermeasureofsecurity,youcanalsocommunicateoveranSSLconnection
ifyouuseadedicatedplanfromMongoDBInc.
With a dedicated plan, you also define custom firewall rules so that your
databaseaccessislimitedtospecificIPaddressrangesand/ortospecificAWS
EC2securitygroups.
DNSandcertificatesetup
IntheIntroduction,yousawaphysicaltopologydiagram(seefigure4)that
showedadomainnamebeingroutedthroughaDNSserverthathadacertificate
toenableHTTPS.Youcannowgoaboutsettingthatup.
First, go to the Route 53 service management console. Click Register
Domain to start the process of getting your own domain, or you may need to
clickGetstartednowundertheheadingtodoadomainregistration,thenclick
RegisterDomain.Ifyoualreadyhaveyourowndomainnameandwanttouse
it, you can do that but that is not covered in this book. Here is the Route 53
managementconsole:
Figure62-Route53servicemanagementconsole
Youcantypeindifferentnamestotryandyouwillbeabletofindonethatis
available.There is an initial chargeas well as a small recurring fee when you
purchaseadomainname.ForthisimplementationoftheNewsWatchersample
app, I settled on newswatcher2rweb.com since newswatcher.com was already
taken.
Figure63-Route53domainavailability
It can take an hour or longer before everything is ready with your new
domainname.Onceitisready,youcangetacertificateallsetupthatusesthe
AWSCertificateManagerService.NavigatetoCertificateManagerandclick
GetStarted.Enterthedomainnamevariationsyouwanttobesupported.You
canusewildcardsandhavetheflexibilityyouneed.Hereisthepagewherethat
happens:
Figure64-AdddomainnamestoAWSCertificateManager
Afterthisscreen,thereareafewmoretoclickthrough.Atsomepoint,you
will also need to validate this action through an email you will receive that
verifiesthatyouaretheownerofthedomainnamethatthecertificateisbeing
setupfor.
AcertificatecannowbesetontheElasticBeanstalkloadbalancer.Youdo
thisthroughtheAWSmanagementportal.Herearethesteps:
1. Open your AWS Elastic Beanstalk environment and on the left,
findandselectConfiguration.
2. Click the gear by Scaling and change Environment type to be
Loadbalancing,autoscaling.
3. Wait for this change to take effectand then click Configuration
againfromtheselectionsontheleftandthenclickLoadBalancing.
4. SelectyourcertificatefromtheSSLcertificateIDdropdown.
5. ClickApplyatthebottomofthepage.
Nowyoucanwatchthestatusthereuntilitindicatesthattheconfiguration
changeissuccessful.
Figure65-ElasticBeanstalkstatus
NowyoucansetuptheDNSroutingtoyourElasticBeanstalkloadbalancer.
Go back into the Route 53 management console and click under the DNS
managementtheHostedzones link. Then click on your Domain name. Click
CreateRecordSet.
Figure66-AWSCertificateManagementCreateRecordSet
Fillouttheformontherightasshowninfigure69.ForAliasTarget,select
yourloadbalancer.Youcanaddarecordsetwithname“www”andthenmake
onewithablankname.
Figure67-Route53createrecordset
Youarealmostdone.YouneedtoturnoffHTTPaccessattheloadbalancer.
Todothat:
1. Go to the EC2 service management console and click 1 Load
Balancers.
2. ClicktheListenerstab,thenclickEdit.
3. DeletetheHTTPentry,thenclickSave.
Figure68-EC2serviceListenerstab
BackattheElasticBeanstalkenvironmentpage,ifyouclickontheURLat
thetop, itwill nolonger work.In yourbrowser, changethe URL tostart with
https:andyouwillbetoldthecertificatedoesnotmatch.NowtypeintheURL
tothedomainnameyouregisteredandyouwillseeeverythingupandworking.
HTTPSisnowworking.Ifyoutry,https://www.newswatcher2rweb.com/itwill
work,buthttp://www.newswatcher2rweb.com/willnotwork.
12.16Deployment
Atthispoint,youhaveeverythinginplacetostarttryingoutyourmiddle-tier
webserviceAPIthatisimplementedasanHTTP/Restendpoint.Youobviously
would not build a service like this without testing it along the way. All of the
discussioninvolving thetesting of theservice is discussedin the nextchapter.
You can see it deployed at this point and then understand the testing that is
necessary.
Don’tgetthewrongimpression.Icertainlywrotethecodeinsmalliterations
andtestedeach andeverybit ofitalong theway.Good developersiterateand
testeverythingastheygo.
At this point, you can zip up your code and deploy it up to AWS. Before
doingso,youcantestthingsoutwithsometeststhatwillbedescribedinchapter
13. You will actually want to test on your local machine as well as deploy to
someknownstagingcloudlocationandtestthereaswell.Youwouldberunning
thetestcodefromyourlocalmachinetogoagainstastagingsitethatishostedin
AWS.
HerearethefoldersandfilesIselectedtohavezippeduponmyWindows
machine.Thereisaright-clickselectiontocreateazipfileunderSendtoona
windowsmachine.
The build and public folders are not actually there yet, so leave those out.
TheyarelaterfoldersthatgetaddedthatholdtheReactapplication.
Youfollowthesameinstructionsdetailedpreviouslytogetyourapplication
code up and running. And now at this point have secured the traffic through
HTTPS.
Figure69-Selectionstomakeazipfile
Note:Ifyoufinderrorsupondeployment,youcanclickonthelogsselection
intheUIoftheElasticBeanstalkAWSconsole.Fromtheactionsmenu,selectto
downloadthefulllogs.Inthefolderyoudownloadyouwillfindafilenamedeb-
activity.log.Scrollandseeifthereareanyindicationsastowhythenpminstall
commandfailed.ThereseemstobeanongoingissuewithAWSElasticBeanstalk
installswheretheyrunoutofmemorywhentheinstallhappens.Toworkaround
this,switchtoat2.smallmachineandthentryyourinstallagain.
Chapter 13: Testing the NewsWatcher Rest
API
NowcomestheexcitingpartwhereyouwillgettoseetheHTTP/RestWeb
APIexercised.Youwillprovethattheserviceisupandrunninglocally,thentest
the deployment and verify everything in production. Once sufficiently proven,
youcanstartthefinaltaskofcreatingaUIforNewsWatcherandbeconfident
thattheintegrationwillgosmoothly.
Thischapter will present several practices thatyou will want to follow for
exercisingyourcodetofullytestit.Itisobviouslyalotsimplertotestanddebug
issueslocally.Let’sfirstlookathowtoemploytechniquesfordebuggingissues
inproduction.
13.1DebuggingDuringTesting
Let’sfirsttalkaboutdebuggingtechniques.Youwillneedtodosomeofthat
asyouruntestsandneedtoexaminetheexecutionofyourcode.
In some cases, output logging to the Node console will provide you with
enough clues to track down an issue. This means that you must log important
thingsthat are happening in theapplication. Beyond this, you willneed a few
toolstohelpyoudoyourinvestigations.
One tool at your disposal is the VS Code debugger. Before deploying
anything,you can run yourcode locally. Youcan useVS Code to debug your
Node.jsprojectcode.
IfyouwanttodebugyourNode.jscode,youopenyourproject,andlaunch
theNode.jsprojectbypressingF5.Youcansetupyourbreakpointsinadvance,
oraddonesasneededthatyouwanttobehit.Onceyourprojectisrunning,you
run Mocha from the command line and exercise your code through tests you
havewritten.Thenyoucanstepthroughyourcode.
Note:Therearesomeissueswhenforkingasecondprocesswhenrunningin
debug mode. That is why I have the line you can uncomment that makes it
possible. Uncomment “var node2 = cp.fork('./worker/app_FORK.js', [], {
execArgv:['--inspect=9229']});”.
TosetupdebugginginVScode,clickthedebugicon.Youwillseeagear
iconat thetop ofthe windowthat youcan clickto createthe launch.jsonfile.
ThisisthefilethatinstructsVSCodehowtoproceed.Youcansetituptohave
twoconfigurationsinthefile.Onewillbeforlaunchingyournodeprocesswith
debuggingcapability.TheotherentryisforattachingtoanalreadyrunningNode
process.Whenyouclickthegearicon,selecttheNodeselectionandcreatethe
file.Yourfilemightlookasfollows:
{
"version":"0.2.0",
"configurations":[
{
"type":"node",
"request":"attach",
"name":"AttachtoRemote",
"address":"TCP/IPaddressofprocesstobedebugged",
"port":9229,
"localRoot":"${workspaceFolder}",
"remoteRoot":"Absolutepathtotheremotedirectorycontainingtheprogram"
},
{
"type":"node",
"request":"launch",
"name":"Launch",
"program":"${workspaceFolder}/server.js"
},
{
"type":"node",
"request":"attach",
"name":"Attach",
"port":9229
}
]
}
Totheleftofthegearisthestartbutton(youcanalsopressF5).Thedrop-
down menu will show you the config section options that come from your
launch.jsonfile. The far right greater-thansymbolin the box opens the output
console window. It is a good idea to always have that open to view any
statementsorerrorsthatgetdisplayed.
Figure70-VisualStudioCodedebuggin
Youcanopenthelaunch.jsonfileandlookatit,butyoudonotneedtomake
anyadjustments to it as the defaultsare just what you need, unlessit is set to
startnodewithanotherfile.Youcanlookupthedocumentationonthesettings
that can be used in the launch.json file on the Node website and VS Code
Website.
Toplaceabreakpoint,openaJavaScriptfileandclickouttotheleftofthe
margin, or click on the line and press F9. Once you hit a breakpoint while
runningcode,yougetfullaccesstoinspectthecallstackandvariables.
13.2ToolstoMakeanHTTP/RestCall
YouwillprobablywanttouseatooltomakeindividualcallstoyourAPI.
Thatway,youcantakesmallstepstogeteverythingverifiedbeforeyouthrowa
testharnessintothemix.IhaveinstalledPostman,FiddlerandCurl.exeonmy
machine for testing individual HTTP/Rest calls. There are many of these such
tools.Withthesetools,youcansendHTTPrequeststoyourserviceandviewthe
returnedresponse.
Fromoneoftherecommendedtools,youcansetuptheverbusage,aswell
astheheadersandtheJSONbodycontent.Youcandoasendandthenlookat
thereturnedresponse.Let’slookattheUIofFiddlerandIwillexplainthebasics
ofhowtouseittocallyourAPI.
YoucansendanHTTPrequesttotheroutehandlerthatregistersauser.This
is an obvious first place to start. To do that, you know you have the
app.use('/api/users',users)callinyourserver.jsfilethattakesyoutotheusers.js
function of router.post('/', function(req, res, next) {}. This is the code that
createsanewregistration.
YouwillwanttomakeapostverbcallandpassinaJSONbodythatcontains
thedisplayname,emailandpassword.Youthenexpecta201returncodetobe
givenback to indicate a successfulcreation. In the return responsewillbe the
returneddocumentthatyoucouldinspect.
YoucannowstartupyournodeservicewithyourprojectopenandpressF5
todebug,orpressCtrl+F5torunwithoutdebugging.Thefirstthingyounoticeis
that you get a console app window that opens in the VS Code UI. That
represents your node process running and the executing of your server.js file.
Youwillseeallofyourconsoleloggingappearinthiswindow.Ifyoudon’tsee
it,clickonthetoolbaronthelefttogettothedebugareaandyoucanopenit
fromthere.
YourNode.js server is running as a local process now. To make an HTTP
requestyoucanconnecttolocalhostwiththeportnumberof3000andinteract
withtheRestAPI.YoucanopenFiddlerandtrythisout.Thefirstthingtodois
tolimitwhattrafficFiddlerseesgoingbackandforth.Otherwise,youwillget
lostinthestreamoftraffic.Todothisandalso makeanHTTPrequestdothe
following:
1. IntheFiddleWebDebugger,clicktheFilterstab.
2. Inthetopsection,checkUseFilters.
3. In the Request Headers section, check Show only if URL
containsandtypein“localhost”.
Figure71-Fiddlerfiltering
4. Clickthedrop-downmenuX->Removeall to get rid of all old
traffic.
Figure72-Fiddlerremoveall
5. ClicktheComposertab,thenclicktheOptionssub-tabunderthat
andmakesureitlooksasfollow:
Figure73-Options
YouwantFiddlertofigureoutthecontentheaderlengthvalueforyou.
6. ClicktheRawtab,andenterthefollowingrequest:
POSThttp://localhost:3000/api/usersHTTP/1.1
User-Agent:Fiddler
Host:localhost:3000
Content-Type:application/json
Content-Length:85
{
"email":"bush@sample.com",
"displayName":"Bushman",
"password":"abc1234#"
}
Figure74-FiddlerRAWrequest
7. ClickExecute.
Ifyoulookintheleft-handpane,youwillseetherequestbeingsentanda
responsereturned.Ifyoudouble-clickthe201response,youcanexamineitand
seethatitworked.ItshouldlookasfollowsifyouclickonRaworJSONview:
Figure75-Fiddlerrequestandresponse
You have now successfully seen your API exercised. Open the MongoDB
Compass tool and you can also see that there is a new document created. In
Compass,youwillseesomethinglikethefollowing:
Figure76-Atlasmanagementportalwithanewdocument
Youcan now enter every single request through Fiddler to prove them all.
Next,trytologinandgetatokenback:
Figure77-Fiddlertologinandgetatokenback
Lookattheresultandyouwillseeatokenintheresponseifyouopenitup.
Youwillneedthistokentouseinthenextrequests.Youcannextmakeacallto
retrieve the user document and see if there are any news stories that have
matchedthefilter.
Figure78-Fiddlertoseethetoken
InFiddler,ClickExecuteandyouwillseearesponsewiththereturneduser
documentandpossibly somenews stories.Thisis greatprogress, andyou can
continuetotryallofyourAPIcallsanddebugeachoneifneeded.Asyouget
eachoneworking,youcancreateatestcaseforeach.
As was mentioned, you might find a tool other than Fiddler that you like.
Curlisa niceoptionbecause youjustopen acommandprompt andexecute it
there.Ithasasimplesyntaxthatyouusetoformulateyourrequests.
13.3AFunctionalTestSuitwithMocha
NowyouwilltakethenextstepofautomatingthetestingoftheAPIusinga
test suite. This will then become your functional test pass. You can start by
implementing some of the same operations that were already tried when you
used Fiddler. To start with, you need to add a new folder to your VS Code
projecttoholdthetests.Youcanalsosetuptheaddednodemodulesyouneed
forthetests.
Note: NPM lets you download different types of modules. Like previous
modulesI’veshownyou,youdownloadthemandthenusearequirestatementto
utilizethemascodemodules.Others,likeMocha,arenotcodemodules,butare
command-line tools. What you do is write a JavaScript file that Mocha will
interpretandrun.Youthenuseacommand-linetoexecuteMocha,anditdoesits
work. Mocha is installed as a local part of your project. This means the
executable will be referenced from that location. Mocha can also be installed
globallyifyoulike.
Addthefollowingtothepackage.jsonfile:
"devDependencies":{
"mocha":"^3.4.2",
"supertest":"^3.0.0"
}
The devDependencies section is reserved for non-production modules that
youwillnotneedtodeploytoaproductionbuild.Theyareonlyneededtorun
yourtestcodelocally.
We cannow go throughsome of thefiles in thetest folder tosee how the
NewsWatcherapplicationcanbetested.ThisfirstfilewillbeusedtohittheAPI
endpointtoexercisetheroutes.
Writingmochatests(functional_api_crud.js)
You will make use of the supertest module and the assert module. The
necessaryrequire()callscanbesetupforthose.Theserverobjectalsoneedsto
be retrieved and given time to be initialized. There are special blocks of code
thatmochawillrunbeforeandafteralltests.Therestofthecodemakesuseof
mochatestblockstoruntestswith.
Therequestobjectiswhatyousetupfromthesupertestmodule.Withthat,
youmaketheHTTPpostverbcallandregisterauser.
You will use the describe keyword for any major test block and use the
individualitkeywords for each test inside of that. With just one test to run, it
wouldlookasfollows:
varassert=require('assert');
varapp=require('../server.js');
varrequest=require('supertest')(app);
describe('Usercycleoperations',function(){
//Waituntilthedatabaseisupandconnectedto.
before(function(done){
setTimeout(function(){
done();
},5000);
});
//Shuteverythingdowngracefully
after(function(done){
app.db.client.close();
app.node2.kill();
app.close(done);
});
it("shouldcreateanewregisteredUser",function(done){
request.post("/api/users")
.send({
email:'bush@sample.com',
displayName:'Bushman',
password:'abc123*'
})
.end(function(err,res){
assert.equal(res.status,201);
assert.equal(res.body.displayName,"Bushman","Nameofusershouldbeasset");
done();
});
});
});
Note:Mochaactuallysupportsseveraldifferentstylesofsyntax,sodon’tbe
confusedifyouseeotherprojectsusingMochaanditdoesnotlookexactlylike
thecodeinthisbook.
Iwillgothroughthepreviouscodetomakesureyouunderstandit.Tostart
with,youhavetheusualNode.jsrequirestatementsatthetop.Forthesupertest
module,youspecifythatyouaregoingtohitthepassedinwebserverendpoint
ofyourNodeproject.Thestringsyousetasparameterstodescribe()andit()are
printed out for you as part of the test run. You should use text that helps you
rememberwhatyouaretesting.
Withthiscode,youhaveatestthatwillverifythatyoucanregisterauser.
Theitblocktakesastringtodescribewhatyouaretestingandthenafunctionto
run.Thereisadone()functionthatyoucalltosignaltomochathatitcanmove
onto the nextit block. Thesetests are each runsequentially. If youdon’tcall
done(),thetestwilleventuallytimeout.
Tousesupertest,youspecifyaverboperationtouse.Thisoneisusingpost.
Youstringtogetherthefunctioncallssend()andend().Eachonewillgetcalled
in sequence. The sendfunction does the HTTP/Rest request and has the body
set.
The end() function can get the response and validate the return code and
valuesfromthereturnedbody.Youusetheassertmoduleforvalidations.
Therearesomecaseswhereyouhaveasecondtestthatreliesontheresults
ofthefirsttest.Thiscanbetrickyifthereisdelayedprocessingofthefirsttest
code. You can either stack one test inside the other, or use a JavaScript
setTimeout()calltodelayyoursecondtestrunbyabitandthenhaveitrun.
Forexample,whenauserchangesanewsfilters,thatoperationwillreturn
immediately. This means that the test code will move on to the next test. The
way NewsWatcher works, is that it sends a message to the forked process to
updatethenewsstories.Ifyouwantedtotestachangetoaprofilefilterstring,
youwouldrealizethatitwouldtakeasecondtodothatinthebackgroundand
youwouldputinadelaybeforethenextcodecouldran.
Thefollowingcodeshowsusinga delayofthreesecondsbefore runninga
test:
it("shouldallowaccessifloggedin",function(done){
setTimeout(function(){
request.get("/api/users/"+userId)
.set('x-auth',token)
.end(function(err,res){
assert.equal(res.status,200);
savedDoc=res.body.newsFilters[0].newsStories[0];
console.log(JSON.stringify(savedDoc,null,4));
done();
});
},3000);
});
Hereisamorecompletesetofteststhatregistersauserandthenmakessure
theycanloginandthendeletestheaccounttocleanthingsup.Thetestrightat
thestartistoverifythatapersoncannotloginiftheydon’tfirstregister.
There may be negative tests you can put in place to verify your error
handling code. In this code below, I make use of local valuables inside the
describe block to pass these between tests such as the token variable. For
example, you need to capture the token at sign in time and keep using it on
subsequentrestcallsthatyouaretesting.
varassert=require('assert');
varapp=require('../server.js');
varrequest=require('supertest')(app);
describe('Usercycleoperations',function(){
vartoken;
varuserId;
varsavedDoc;
//Waituntilthedatabaseisupandconnectedto.
before(function(done){
setTimeout(function(){
done();
},3000);
});
//Shuteverythingdowngracefully
after(function(done){
app.db.client.close();
app.node2.kill();
app.close(done);
});
it("shoulddenyunregistereduseraloginattempt",function(done){
request.post("/api/sessions").send({
email:'bush@sample.com',
password:'abc123*'
})
.end(function(err,res){
assert.equal(res.status,500);
done();
});
});
it("shouldcreateanewregisteredUser",function(done){
request.post("/api/users")
.send({
email:'bush@sample.com',
displayName:'Bushman',
password:'abc123*'
})
.end(function(err,res){
assert.equal(res.status,201);
assert.equal(res.body.displayName,"Bushman","Nameofusershouldbeasset");
done();
});
});
it("shouldnotcreateaUsertwice",function(done){
request.post("/api/users")
.send({
email:'bush@sample.com',
displayName:'Bushman',
password:'abc123*'
})
.end(function(err,res){
assert.equal(res.status,500);
assert.equal(res.body.message, "Error: Email account already registered", "Error should be already
registered");
done();
});
});
it("shoulddetectincorrectpassword",function(done){
request.post("/api/sessions")
.send({
email:'bush@sample.com',
password:'wrong1*'
})
.end(function(err,res){
assert.equal(res.status,500);
assert.equal(res.body.message,"Error:Wrongpassword","Errorshouldbealreadyregistered");
done();
});});
it("shouldallowregisteredusertologin",function(done){
request.post("/api/sessions")
.send({
email:'bush@sample.com',
password:'abc123*'
})
.end(function(err,res){
//<Session&Cookiecode>cookies=res.headers['set-cookie'];
token=res.body.token;
userId=res.body.userId;
assert.equal(res.status,201);
assert.equal(res.body.msg,"Authorized","MessageshouldbeAUthorized");
done();
});
});
it("shouldallowregisteredusertologout",function(done){
request.del("/api/sessions/"+userId)
.set('x-auth',token)
.end(function(err,res){
assert.equal(res.status,200);
done();
});
});
it("shouldnotallowaccessifnotloggedin",function(done){
request.get("/api/users/"+userId)
.end(function(err,res){
assert.equal(res.status,500);
done();
});
});
it("shouldallowregisteredusertologin",function(done){
request.post("/api/sessions")
.send({
email:'bush@sample.com',
password:'abc123*'
})
.end(function(err,res){
token=res.body.token;
userId=res.body.userId;
assert.equal(res.status,201);
assert.equal(res.body.msg,"Authorized","AuthorizedMessage");
done();
});
});
it("shoulddeletearegisteredUser",function(done){
request.del("/api/users/"+userId)
.set('x-auth',token)
.end(function(err,res){
assert.equal(res.status,200);
done();
});
});
});
You run your Mocha test suite from a command prompt. On Windows
machines you open a Node.js command prompt. You can also use GitHub
DesktopandrightclickyourrepositoryandselectOpeninGitshell.Onceyou
areatthecommandprompt,navigatetothelocationofyourprojectifneeded.
ThisisnotnecessaryifyouopenupapromptfromGitHubdesktop.
Youdon’tneedtostartupthenode.jsapplication,becausetheMochacode
will do that. You can run Mocha from the local project folder in the git shell
windowasfollows:
./node_modules/.bin/mocha--timeout30000test/functional_api_crud.js
Youmayneedtoplayaroundwiththetimeoutargument.Itispossibletoget
falsefailuresbecauseofMochatiming-outand movingontothenext itblock
tooquickly.Theoutputforthecompletefunctionaltestsuitelooksasfollows:
Figure79-Mochafunctionaltestoutput
Note: You should realize that, even if you are running against your local
Node.js service, you are still hitting the real AWS hosted MongoDB database.
YoumightneedtoopentheCompassappanddeleteunwanteddocumentsthat
youcreatedthroughyourtests.ThetestsincludedwiththeNewsWatchersample
are written to clean up after themselves. Of course, you could install a local
copyofMongoDBonyourmachineandconnectagainstthataswell.
TorunagainstthedeployedNode.jsapplicationinAWS,youcanchangethe
supertest usage to go against the production URL, such as
https://www.newswatcher2rweb.com. You can see in the GitHub project that I
haveafewcommentedoutlinesfordifferentwaystorunthetests.
13.4PerformanceandLoadTesting
Writinganapplicationthatcanserveasingleuserisnotabigchallenge.The
realchallengecomeswhenmultiplepeopleareallhittingthewebserviceREST
APIatthesametime.
To write the NewsWatcher sampleapp and get it towork for a singleuser
wasjusttwoweeksofworkforme.Togetittoscaleandhandlethesimulated
loadof many users, requiredmuch longer than thatto work out allthe issues.
You certainly don’t want to wait until your big production rollout to find that
yourcodefallsflatonitsfacewhenmorethanonepersonusesit.
Howareyougoingtoaccomplishtestingyourcodeatscale?Theonlyway
to accomplish this is with a test suite that can provide usage in parallel and
simulatemultipleusers.
ThereareUItestingtoolsthatcanrecordyourusageofawebsiteandreplay
it. They can be replayed more than once at the same time to simulate lots of
interaction happening. This might work well for some sites that serve static
contentand have no concept of people logging in and exercisingsome unique
workflowinthebackendservicelayer.
Loadtestingenablesyoutoprovethescalingofyourapplication.Youalso
use this to measure your SLA (Service Level Agreement) values under a
constant load. You can experiment by increasing the load until you find the
breakingpoint.Thiswilltellyoutheabsolutepeakvaluesyoucanrununder.To
dothis,youneedtoaltertheURLtobethatoftheproductionorcloudstaging
environment.
Running the load testing suite can also help you test out your Elastic
Beanstalkscalingstrategiesandtopology.Itwillofcoursebeusefulinverifying
theperformanceofMongoDBthatyouhaveworkedhardtooptimize.
13.5RunningLint
You can install the npm packages "eslint" and "eslint-plugin-react". These
allow you to run validations that scan the code and look for syntax errors,
uninitializedvariablesandevenspecificstyingthatyouwant.Iwouldsaythatit
isarequirementtohavethisinplace.Ifyouwerewritinginalanguagesuchas
C++thatwascompiled,yougetthatalldoneupfront.Unfortunately,youwrite
codewithJavaScript,butthendon’tgettoseeerrorsuntileachlineexecutes.
Youwillseethatthepackage.jsonfilehassomescriptssetuptorunlintand
alsocombinetherunningoflintwiththerunningofthefunctionalmochatests.
Thisiswhatisinthepackage.jsonfileforthat:
"scripts":{
"lint":"eslint**/*.js",
"test":"mocha--timeout30000test/functional_api_crud.js",
"pretest":"npmrunlint",
"posttest":"echoAlltestshavebeenrun!"
}
Whenyoufirstinstalleslint,youcanrunitininitializationmodeanditwill
createafilebasedonthequestionsyouanswer.Thecommandis:
./node_modules/.bin/eslint–init
Iranthisandthenmadeafewtweakstothefollowingtwofiles:
//.eslintignore
node_modules
build
src
test
Andtheotherfile:
//.eslintrc.json
{
"env":{
"browser":true,
"es6":true,
"node":true
},
"extends":"eslint:recommended",
"parserOptions":{
"ecmaFeatures":{
"experimentalObjectRestSpread":true,
"jsx":true
},
"sourceType":"module"
},
"plugins":[
"react"
],
"rules":{
"linebreak-style":[
"error",
"windows"
],
"no-console":"off"
}
}
It is as simple as running the following commands. The first just runs lint
andthesecondrunslintandthenthetests.Lintwilllistalltheerrorsandyoucan
keepfixingthemuntiltheyareallgone.Herearethecommands:
npmrunlint
npmtest
Chapter14:DevOpsServiceLayerTips
It is a fabulous accomplishment to get the code all tested and deployed to
production. Don’t get too comfortable though, as it is quite another matter to
managetheoperationsofafull-stackapplication.Thischapterwillpresentsome
key skills that will make your life easier when it comes to running a 24x7
operationsforyourservicelayer.
Chancesarethatyouwillexperiencesometypeofcatastrophicfailurebefore
too long. First off, you absolutely want to do everything up front to put
preventativemeasuresinplace.Astheoldsayinggoes“Anounceofprevention
isworthapoundofcure.”
Youalsoneedtoput aplaninplaceforhandlinga crisiswhenithappens.
Youwanttobeinapositiontohavealloftheinformationatyourfingertipsto
make it possible to recover in the shortest amount of time. There are some
generaltechniquesthatwillbepresentedhere,butyouwillhavetocomeupwith
yourownspecificstrategiesthatfityourownenvironment.
Letmemakeabriefcommentaboutcontinuousintegrationandcontinuous
delivery(CI/CD).Ifyouareworkingonanykindofsubstantialprojectthatwill
begoingonforawhile,orhasmultiplepeoplecontributing,youdefinitelyneed
to implement CI/CD. Manually performing the tasks of building, testing, and
deployingcodegetsoldfast.Alwaysrememberthatdoingthesethingsmanually
is prone to human error. If your DevOps process is not automated with full
integrationtesting,thenitisnotreallycomplete.
Thereisawisesayingthatstates“Youhavetoslowdowntospeedup.”You
caninterpretthattomeanthatalittleinvestmentupfrontpayshugedividends
over and over. This ability to centrally coordinate CI/CD really helps groups
withanagileprocessiteratemorerapidly.Productivitygoesupbecauseofthis
automation and team downtime is reduced because integration bugs are not
spreadacrosstherestoftheteam.Itismuchmoreexpensiveintimeandmoney
tocatchbugsinproduction,sothereisahugesavingshere.
14.1ConsoleLogging
Writingmessagestoaconsoleoutputor logfileisanage-old practicethat
mightbeusefulifyoucanavoidbeingoverwhelmedbytoomuchloggingand
thenbeabletointerprettheinformationtosolveproblems.
NodehastheConsolemodulethatisusefultowritetraceoutputto.Thisis
available in your application already, so you don’t need to use a require
statement. Here are some useful methods you can use that are found on the
consoleobject:
log()
info()
error()
warn()
Basicmethodforoutputtingtext.Itcomesinseveraldifferent
formsallwiththesamefunctionsignature.
console.log(“wemadeithere%d”,someVarNumber);
dir()
Itisusefultoviewanobjectyoumighthaveinyourcode.There
areoptionsavailableforthis,forexampletorecursefurtherthanthe
defaultdepthof2levels.
console.dir(someObject);
time()
and
timeEnd()
Tologelapsedtime,youusethesetwomethods.Youwillgetthe
elapsedtimewhenyoudothefollowing:
console.time(“start”);
//somecodeoperationsthatyouwanttotime…
console.timeEnd(“start”);
trace()
Forshowingastacktracefromthepointinyourcodewherethis
iscalled.
console.trace(“someLabel”);
assert()
Thisisthestandardassertionusagecommonlyavailable.
console.assert(valid,“Hi”);
You can set up logging to go into a file that can be looked at. Sometimes
loggingwillpointyouinthegeneraldirectionandthenyoucanusethedebugger
to further diagnose an issue. As another option you could send every log
message to a special collection in MongoDB. You can set up MongoDB
documentstohaveatimetolive(TTL)beforetheyareautomaticallydeletedor
setthecollectionascapped.
14.2CPUProfiling
TheV8enginecanprovideyouwithCPUusagereports.Thesimplestwayto
dothisistolaunchyournodeprocesswiththeprofileflagset,afterrunningyour
testcode,youstoptheprocessandyouwillthenhaveafileavailabletoview.
YouwouldlaunchNodeasfollowsonyourlocalmachine:
node--profserver.js
Ifyouthenrunsometestcode,suchasyourloadtestingsuite,youcanget
some idea of where your code might be spending most of its time. Once you
have run your test code for a bit, you stop the node process and a file named
somethinglike“isolate-000001D213C4D490-v8.log”willbesaved.
Next, open your browser and navigate to the Chrome V8 profiling log
processor site at http://v8.googlecode.com/svn/trunk/tools/tick-processor.html.
Youcanopenyourlogfilewiththistoolandseeanice,crisplayoutofyourcode
callslistedinorderofwheremostofthetimewasspent.Thereisacommand
lineversionofthetickprocessoralsoavailablefordownload.
Youwillseereportsthatlookasfollows:
Figure80-V8Reportsample
Figure81-V8Reportsample
You can also utilize the V8-profiler using a Node module to start it up in
yourcodeifyousetuproutehandlersandwanttoprofilethingsinproduction.
TheV8engine,whichisoneofthecomponentsthatNode.jsisbuilton,letsyou
run an analysis of production running code. Profiling data can be sent to an
externalfilethatyoucanlateropenandviewtheresultswith.Hereisthecodeto
accomplishaprofilerun:
varfs=require('fs');
varprofiler=require('v8-profiler');
profiler.startProfiling();
...dosomeprocessing...
varprofileResult=profiler.stopProfiling();
profileResult.export()
.pipe(fs.createWriteStream('profile.json'))
.on('finish',function(){
profileResult.delete();
});
Here is a screenshot of the Chrome Dev Tool (press F12 while running
Chrome)thatiscapableofreadingtheprofilefile.
Figure82-ChromedevelopertoolV8profilereport
Toreadinyourfile,ClicktheProfilestab,thenclicktheProfilespaneand
clickLoadtoloadyourfile(s).
Withprofilingturnedonwhilealoadtestwasrun,Iwasabletoproducea
reportthatshowedafewissuesIcouldaddress.Oneofthemturnedouttobethe
useofbcryptfor password hashing. As seen in the profile analysis, you see a
bottom-upviewofthecallingtree.Ifyouexpanduntilyouseeyourfunctions,
youcanthenunderstandwherethecallisbeingmadefrom.Here,youseethat
thebcrypt.hashSync()functioncalltook19%ofthetime.
Figure83-Profiledrill-down
Hereisthecodeforwhatyouseelistedforthefirstissue:
findUserByEmail(req.db,req.body.email,functionfindUser(err,doc){
...
passwordHash:bcrypt.hashSync(req.body.password,10),
Itwasobvious,giventhenameofthefunction,thatIwasnotusingtheasync
version of the hash-generation function. Go figure, I should have known that
hashSync() was not a good idea to call. As you know, you want to minimize
CPUusageonthemainNodethread.Checkingthedocumentation,Ifoundthe
asyncversionandmadethechangetouseit:
bcrypt.hash('bacon',8,function(err,hash){
});
Note:Itishelpfultogivenamestoallanonymousfunctions,otherwiseyou
willjust see a lot of “anonymous” functions and it is harder to pinpoint what
functionsareintheprofilelisting.
14.3MemoryLeakDetection
Inamanagedlanguageframework,youdon’tdirectlyallocatememoryand
subsequentlyfreeitup.Youcandoanewanddeleteofanobject,butyoustill
don’thavecontroloverthatmemory,suchastheactualreclaimingofit,ordoing
thingslikehavingmemorypointersintoit.Instead,thereisagarbagecollector
thatkeepstrackofmemoryreferencesandtheGCdecideswhentorunandwhen
memorycanberecycled.
The truth is, that with garbage collection running, you can still run into
memoryleaksthatwilleventuallycauseyourapplicationtoeitherrunslowly,to
completelyfreeze,orcrash.
Youcanobviouslycreateamemorygrowthproblemifyouhadsomethingas
simpleasanarraythatyoucontinuallypusheddataintoandneverfreeup.Ifyou
held on to an object reference permanently after you no longer needed it, the
garbagecollectorwillnoteverreclaimit.Athoroughcodereviewofcallbacks,
closures, constructor functions, and arrays can be a starting point to finding
memoryleaks.
Ultimately,yourbrainmightnotbeabletotracethroughalloftheintricacies
ofyourcodeandyouwillneedtotakememorysnapshotsthatcanbecompared
acrosstime.Node.jsapplicationsarealwaysbuiltwithseveral,ifnotdozens,of
downloadedmodules.Youhavetobesuspiciousofthose,aswell,astheymight
containmemoryleaks.
Youcan watchthe OS reporting of memory for your Node.js process over
time and see what kind of graph you have and, if you see an ever-increasing
amount of memory being taken up, you can then dive in and investigate. It is
even not unheard of for people to resort to restarting their Node.js processes
everydayjusttocircumventanymemoryleakproblems.Youmight,inreality,
nothavealeakif,overtime,youcanseethegarbagecollectorkickinanddoits
job.Comparememorysnapshotsovera24-hourtimespan.
You can use the v8-profiler module that was previously used for CPU
profiling to take memory snapshots. You can likewise have the output files
viewedintheChromedebugger.
varfs=require('fs');
varprofiler=require('v8-profiler');
varsnapshot=profiler.takeSnapshot();
snapshot.export()
.pipe(fs.createWriteStream('snapshot.json'))
.on('finish',snapshot.delete);
Note:Rememberto neverrunyour CPUprofiling atthe sametimeas you
take heap snapshots. The overhead memory usage for the CPU profiler will
inundateyou.
I have set up a specific route through Express that can trigger a memory
snapshot.ThatwayIcaninsertthisabilityintomytestsandhaveitavailablefor
measpartofaloadtestingrun.Hereisascreenshotofamemorysnapshotfile
thatisloadedintotheChromebrowserdebugger.
Figure84-Chromedevelopertoolwithmemorysnapshot
The Distance column shows you how many steps removed from the root
objectthememoryreferenceis.Youcanusuallyassumethattheobjectwiththe
shortest distance is the one causing a memory leak. The Shallow Size is just
whatthisoneusageistaking.RetainedSizegivesyouallofthespacethatwould
befreed up thatthis object isreferencing and thus holdingon to. Thisis only
trueifthoseobjectsarealsonolongerreferencedbyanythingelse.
Therearedifferentviewsyoucantryout,suchasContainment,whichhelps
youalsoviewlow-levelmemoryinternals.
14.4CI/CD
Doyouworkonaprojectwherethetasksofbuilding,testinganddeploying
codearealldonemanually?Whyisthat?Frommyexperience,thereasonteams
don’tautomate manual tasks is typically because they either lack incentive, or
they lack the knowledge. Lacking incentive is really a poor excuse. There is
plenty of incentive if you honestly look at the return on investment of
implementingaCI/CDprocess.Ifyour DevOpsprocessisnot automatedwith
fullintegrationtesting,thenitisnotreallycomplete.
Ionceheardaspeakerataconferencesaytodevelopers–“Itain’tdoneuntil
itisautomated”.Hewassayingthatyouneedtoautomateeverythingfromthe
check-in,testingandintegrationthroughbuildinganddeploying.Inthisbook,I
willsaythe‘D’inCDstandfordelivery.Thismeansthatcodeisservedupand
readytobedeployed.Wewill,however,taketheprocessallthewaythroughto
deliveryandcansayweimplementedaCI/CD+solution.
AnotherstatementIhaveheardandthatIoftenrepeatisthatof“Youhave
toslowdowntospeedup.”Youcaninterpretthattomeanthatalittleinvestment
upfrontpayshugedividendsoverandover.Thisabilitytocentrallycoordinate
CI/CDreallyhelpstheagileenvironmentiteratemorerapidly.Productivitygoes
up because of the automation and team downtime is eliminated because
integrationbugsarenotspreadtotherestoftheteam.Itismuchmoreexpensive
intimeandmoneytocatchbugsinproduction,sothereisahugesavingstobe
hadhere.
Nowthatyouhavetheincentive,let’sassumethatitreallycomesdownto
knowledge.Afterreadingthis,youwillbeinformedandwillnolongerhaveany
moreexcusespreventingyoufromimplementingCI/CD.
ManytoolsandwaystoaccomplishCI/CD
Implementing CI/CD can, of course, be a complex undertaking. No one
articleor bookcan tellyou everythingyou need toknow becausethere areso
manytoolsoutthere.Noonesolutionwillworkforallprojects.Allprojectsare
unique in their code structure and frameworks used. We will only scratch the
surface and only do so for a Node.js project in a very narrow niche. We also
narrowourfocusdownbyrelyingonaPaaSsolution.
IfyouchoosenottogotherouteofsettingupacompleteCI/CDtool,you
can still automate all of this on your own and launch things manually. For
example,youcanimplementaseriesofGulptasksthatwoulddoalotofwhata
CI/CDsystemwoulddo.
Note: You can do everything in your DevOps environment through the
commandline.Onceagain,IhavechosentotaketherouteofusingUIswhere
available.
Youmusthavedevelopertestedcode
As a preparatory step, a developer will have prepared some code and
privatelytestedanynewfeaturebeforesubmittingittotheCI/CDprocess.The
testing should be as thorough as possible in the context of the rest of the
integratedsystem.Ifthatisnotpossible,thenitcanbetestedasanisolatedunit,
perhapswithstubbedoutormockedfunctionalityinjected.Thetestingshouldbe
runnablebyadeveloperbytyping“npmtest”,whichwillrunlintandthenall
testsforthatpartofthecode.
Once the developer has had their own private verification completed, they
can do their code check-in that will then be staged to go into the pipeline for
consideration.IfyouareusingGitandGitHub,thecodewouldbepushedtothe
branch you designate for the CI/CD to be triggered from. You would not
necessarily push it into the master branch. The master branch should only be
usedforthecodethatisrunninginproduction.Youcouldpushtomasterandtag
thatbutnothaveitdeployedtoproduction,buttotestandthenhaveastepthat
requiresmanualacknowledgmenttopushthattoproduction.
CI/CD is all about increasing your iteration speed and the quality of
everythingwritten.Ofcourse,youhavetoprovidehigh-qualitycomprehensive
testsuitestoachievethis.Onceyourcodeisreadytocommit,thenthepipeline
workflow of CI/CD takes place. So what does the cycle look like? Let’s go
througheachofthesteps.Hereisasimplediagramtoshowyouthepiecesthat
wouldmakeupasimpleNode.jsCI/CDsystem:
Figure85-???
AWStools
TherearealotofcompaniesmakingtoolstohelpwithdoingCI/CD.Ihave
lovedusingtheopensourcetoolJenkinsinthepastanditisagoodchoice,and
thereareevencompanieshostingitinaPaaSenvironmentifyoudon’twantto
installandmanageityourself.
This book will use AWS CodePipeline as the overall orchestration tool to
mage the build, test and deployment steps. The build and test are done with
AWSCodeBuild.DeploymentcanbedonewithCodeDeploytodeploytoECS
foraMicroservicesarchitecture.Inourcase,weuseElasticBeanstalk.Thereisa
directintegrationtothatinCodePipeline,soCodeDeploywillnotbeused.
StepOne:Install
Once the code is committed into the source code repository then the code
needs to be staged in the overall application. CI/CD tools actually are able to
listen to GitHub and will kick off immediately upon seeing a code push. The
ideaisthataCI/CDsystemcanseethepushanddoalocalcloneofthebranch
soyouhave afreshlyintegrated codeversionto nowwork frominthe CI/CD
system.
AllcodedependenciessuchasNPMmoduleswouldneedtobeupdatedin
thiscloneandthatispartofwhattheCI/CDwouldkickoff.Thisisdonewithan
npminstallcommand.Sincewearewantingtogetaproductionenvironmentup
andrunning,wewillhavetodoanextranpminstalltogettheDevDependencies
sincethosedon’tgetinstallediftheenvironmentvariableNODE_ENVisequal
to“production”.Thiswillbeexplainedagainaswegothroughthesteps.
StepTwo:Build
Onceallthedependenciesareinstalled,wecandoanybuildandteststeps.
In our case, the Node.js code is set to go, but there would be a build step
required for our React code. Then things like Lint, can be run. If Lint passes,
thentheMochatestscanberun.
ComprehensivetestingiscentraltogettingCI/CDworking.Thisstepcould
runathoroughsuiteoftestsatalllevels,suchasUnit,integration,feature,load,
performance and UI automation testing. A perfect pass is expected and you
would get generated reports to show that all went well and also produce code
coveragereportingandhavememoryleakdetection.
StepThree:Post-Buildanddeploy
This is the final step that can generate the files necessary to deploy and
actuallydothedeploymentifyoulike.Sincewehaddevelopmentdependencies
likeLintandMochainstalled,wecanbackthoseoutandjusthavedependencies
neededfortheproductionenvironment.Thedeploymentcanbeautomatic,oran
email can be sent to a person to do manual acknowledgment to approve it. If
your team is in favor of TIP (Testing in Production) you could deploy to a
reservedhiddenportionofyourproductionenvironment.
You should have a testing environment that is as close to production as
possible.Onethingtorememberistoalsohaveinplaceadedicateddatabasefor
testing purposes hosted in PaaS that is always available. You do not need to
continuallydeploythereifitissimplythelocationofthedataandnothingneeds
tobepreconfigured.YoucanrunsomecleanupscriptaspartoftheCI/CDstep
here if that is necessary and also some pre-population script if you require
certainDocumentstobeinplace.PaaSofferingsforMongoDBandothersare
availableasdocument-baseddatabases.
Ifthetestingfails,youwouldbenotifiedandtheCI/CDprocesswouldnot
proceedanyfurther.Youwouldwantfailuresautomaticallyenteredintoanissue
trackingsystemtoofficiallytrackandresolveissues.
NewsWatcherCI/CDwithAWSCodePipeline
Here is a quick introduction to how to set up a CI/CD process with AWS
CodePipeline.
MakesureyouarealreadysignedintoGitHub,andthenintheAWSconsole
select CodePipeline from the Services drop down. Here is a brief listing of
someofthesteps.Ihaveleftoutsomeofthedetailsastheyshouldbeobvious.
1. ClickGetStarted.
2. NameyourpipelineandclickNextstep.
3. For "Source provider" select GitHub and click Connect to
GitHub.
4. ClickAuthorizeaws-codesuiteandconfirmyourpassword.
5. Select the Repository and Branch. I selected the master branch.
ClickNextstep.
6. SelectAWSCodeBuildfortheBuildprovider.
7. SelecttheruntimetobeNode.jsandthelatestversion.
8. FortheBuildspecification,selecttouseabuildspec.ymlfile.
HereiswhatIhaveinthebuildspec.ymlfile:
phases:
install:
commands:
-echoInstallingNodeModules...
-npminstall
-npminstall--only=dev
build:
commands:
-echoBuildstartedon`date`
-echoBuildingReactWebapplication
-npmrunbuild-react
-echoPerformingTest
-npmtest
post_build:
commands:
-echoFinalbuild,withoutdevDependencies
-rm-rfnode_modules/
-npminstall
artifacts:
files:
-'**/*'
Once you are ready to go, you can merge and push a code change up to
GitHub and watch in the AWS console with CodePipeline as each step
progresses and finally see the code pushed out to production. Here is a
screenshotshowingthefullprocessafteritisrun.Ifyouhappentohaveanerror
intheCodeBuildstep,thenyoucanopenthatupandseethedetailsandseehow
tofixtheerror.
Figure86–AWSCodePipelineusage
14.5MonitoringandAlerting
TheAWSElasticBeanstalkmanagementportalhasaMonitoringpagewith
whichyoucan viewkey machineperformance metrics.Youdefinitelywant to
opentheportalandlookatwhatisavailabletobemonitored.Notonlycanyou
lookat trending charts,but you canalsoset up alertsthat send you emails,or
even get text messages in the case of thresholds being crossed. Here is the
Monitoringpage:
Figure87-ElasticBeanstalkmonitoringpage
YoucanclickEditandselectwhatyouwanttobegraphedout.Youcanalso
clickthealarmbelliconandcreateanalarmthatwouldnotifyyouifyoucrossed
somethreshold.
Alternatively, you can open up the CloudWatch management console in
AWS and explore similar capabilities for metric viewing, logs, events and
alarms.
Figure88-AWSCloudWatchmanagementconsole
OnethingtoknowisthattheNode.jsprocesscancrashandifthathappens,
it needs to be restarted. In an IaaS environment, you would be responsible to
havesomemechanismtodetectthisandrestartyourNode.jsprocess.
There are NPM downloads such as “forever” and PM2 that do this. With
AWSElasticBeanstalk,itusesNginxandtherestartingishandledautomatically
foryou.ThisisanotherexampleofhowPaaSreallydonerightcanmakeyour
lifeeasier.
If your application is not available, reliable, and performant, then your
customersandyourbusinesswillsuffer.Thegoalofmonitoringandalertingis
tomaintainapplicationavailability,reliability,andperformance.Youdothisby
implementingApplicationPerformanceMonitoring(APM).Thisallowsyouto
discoverproblemsbeforeanyoneelsedoesandthentoachieveresolutionsinthe
leastamountoftime.
TobeginimplementinganAPMstrategy,youneedtoinstrumentyourcode
and surface events, logs, and metrics. Then you use an APM tool to chart out
performancemetricsandsetupalarms.
Youneedmetricssothatyouarenotflyingblind.Apilotcanflyaplanein
the dark because of instrumentation and telemetry. The last thing you want to
havetodoisremoteintoindividualserversandstartpokingaroundtosearchfor
a “needle in a haystack”. Instrumentation and monitoring is the only way you
willbeabletoscaleandsurvive.
AWSX-RayforcodeinstrumentationAPM
AWSoffersX-Rayasawaytoinstrumentyourcodeandhavethetelemetry
viewedinaportalthatletsyouinspecteachofthetracesthatyourapplicationis
processing.Atraceconsistsofinformationforwhatisoccurringinyourservice
layer.Forexample,youcanseeatraceforeveryHTTProuteendpointandeach
oftheverbsthatarebeinghandled.
To get X-Ray tracing to work with your Elastic Beanstalk application you
needtoturnonasetting.GointotheElasticBeanstalkconsoleandthenintothe
Configuration settings for the Software Configuration and check the box to
enable the X-Ray daemon. This will run the agent on the EC2 machines that
collect the data and forward it to the location it can be collected together for
viewingandalerting.ThiswillenableXRaytogetatmachineresources.
Note:NewRelicisatoolthathasbeenaroundforsometimeandisavery
comprehensive solution for gathering machine resource usage and transaction
instrumentation.LikeAWSX-Ray,youcanaddcodetoyourNode.jsprojectto
instrument the sending of telemetry. You can customize what is sent back and
addcustommetricsandevents.YoucanalsoaddascripttoyourHTMLclientto
instrumentfromtheclientsideallusage.NewRelichasarichcapabilitytoset
thresholdsandalertonthem.
TherearecostsassociatedwithusingX-Ray,butifyouhavetoprocessalot
oftracinginordertoincuranycost.PleaselookattheAWSdocumentationfor
anexplanationofthecost.Youcancontrolwhatitactuallycapturesonaroute-
by-route basis. You can completely ignore a route, or set it up to only have a
certainpercentageoftrafficcaptured.
X-Ray understands how to collect tracing information for transactions that
are handled by your Express usage in Node through the aws-xray-sdk npm
module.
Note: To download the module, you need to be running your machine
consoleasadmin.Thenyoucanrun“npminstallaws-xray-sdk–save”.
HereiscodetouseAWSX-RayinNodeJavaScript:
varAWSXRay=require('aws-xray-sdk');
app.use(AWSXRay.express.openSegment('NewsWatcher'));
//AllyourAPIroutes
app.use('/api/users',users);
app.use('/api/sessions',session);
app.use(AWSXRay.express.closeSegment());
IfyouthengotoX-RayintheAWSconsoleyoucanseethetraces.Youcan
alsosee all of the AWS resources in your service map. Here is what thetrace
viewlookslike.Youcanclickonatraceandseeitsdetails.
Figure89–X-Raytrace
If your code has an unhandled exception, then you could actually see the
stacktraceinthedetailsofthetrace.Youcanalsoaddextrainformationtoany
traceasfollows:
varsegment=AWSXRay.getSegment();
segment.addMetadata("errorHandler",err.toString());
IfyouwereusingtheAWSSDKinyourcode,youthenyoucanaddsome
codetomakeitawareofthosecalls,andthatusagewillbeinstrumented.Asof
yet,thereisnosupportforMongoDB,butIanticipatethattherewillbesoon.
Itispossibletoputincodetomeasureanypartofatransactiontracesuchas
someprocessingcodeyouwanttohaveafurtherbreakdownoftoseeitstiming
intheoveralltracetime.Thiscanbesynchronous,orasynchronouscode.
TheideaisthatthereisanincomingrequesttoyourNode.jsservice,suchas
aPOSTtoanAPIendpoint.AsingletraceIDisassignedtothatrequestandno
matterwhathappensuptothevery endofthatrequest,everysubpart ofwork
that goes on shares that ID. That means all of the work in your code, calls to
other backend HTTP services. Any use of other AWS resources is all tied
togetherandviewableunderthatsingleIDintheX-Rayconsole.
PARTIII:ThePresentationLayer
(React/HTML)
Partthreeofthisbookwillteachyouaboutthepresentationlayerofathree-
tierarchitecture.Indoingso,thesampleapplicationwillbeextendedtobringit
to a state where the UI is functional and interacting with the services layer.
Beforereadingthischapter,youshouldobtainabasicunderstandingofHTML.
In this part of the book, I present the technology of React as a framework
thatfitsnicelyintotheoverallarchitectureasawaytobinddatatoandfromthe
service layer web service into a UI. The UI will be rendered as an HTML
WebsiteandalsoasaNativemobileapplication.
If you have followed along in the previous chapters of this book, you will
realizethatthereisaservicelayerbuiltandtestedfortheUItoconnectto.This
is very important, as the Node.js application will perform the dual role of
servicingnotonlytheHTTP/RestAPI,butalsoofservingupyourReactHTML
andJavaScriptfilesetc.toabrowserasaSPAapplication.
Note:ItisnottheintentionofthisbooktobeacomprehensiveguidetoUX
designorSPAwebdesign.Iwillonlytouchonsomeconceptsandthensticktoa
narrowtechnology presentation.There are wholebooks eachdevoted toReact
andReactNative,so youcan understandthatwe canonlycover thebasics of
howeachworktogetyougoingonthistopic.
Chapter15:Fundamentals
I will now go over the fundamental concepts of the top tier in a three-tier
architecture. You will see what capabilities are essential and find a list of
questionsto consider when doing your code design. Youcan then get into the
specificsofReactasaJavaScriptUI frameworkandlearnhowitwillbe used
with the NewsWatcher sample application. One of the reasons that React was
chosen as a technology is because it fulfills the needs of this top layer of the
applicationarchitectureanddoessowiththeJavaScriptlanguage.
15.1DefinitionofthePresentationLayer
Anyapplicationthatrequiresuserinteractionwillneedapresentationlayer.
Humansneedapresentationlayertoviewdataandtoallowthemtoinputdata.
Forexample,anonlinebookstorewouldbepresentingalistofbookstoauserto
browsethrough.Eachbookoffered,mightcontainaphotoofthebookandthe
dataassociatedwithitsuchasthetitle,author,description,publicationdate,and
cost.Inputgatheredfromtheuserwouldbethingslikebookorders,reviews,and
customerservicequestions.
MV*andSPAdesigns
Youneed toemploy thetechniques ofabstraction andcomponentization in
all the layers of an architecture. This is no different with the coding in the
presentationlayer.
Oneofthebenefitsofchoosing aframeworkforyourpresentationlayeris
thatmostframeworksaresetuptoemploysometypeofMV*patternthatlends
itselftoanorganizedsetofcomponentsthatmakeupyourcode.
Make yourselves acquainted with MV* design patterns and with what a
SinglePageApplication(SPA)designentails.Thereareseveralexcellentbooks
available and plenty of online material to study. There are many things about
SPAdesignsthatmakethemagreatchoicetoday.
Note: You need to be aware that if you are concerned about SEO (Search
EngineOptimization)youmightneedtoswitchsomeofyourpagerenderingto
beserver-sideandnotbefullyrenderedasaclient-sideSPA.Thisissometimes
requiredforsearchenginestoindexyoursite.Reactcanbesetuptogiveyou
thebalanceyouneedforthis.Thereisashortchapterdevotedtohowtodothis.
Presentationlayerplanning
Therearemanydecisionsthatgointocreatingapresentationlayer.Thevery
firststepinvolvesplanningforwhattypeofinteractionsyouruserswouldwant
to take. You will want to make an initial sketch of the UI of your application
early on before coding anything up. This would happen concurrently as you
designtheservicelayer.Itiswisetopursueasimultaneousbottom-upandtop-
downapproach.
Knowingwhatoperationsandworkflowsareneededisanotherfirststepin
fleshingoutapresentationlayer.Thefollowingquestionsareusedtohelpyou
determinethedesignofapresentationlayer:
Have you done any of the following? sketching, prototyping,
storyboards, surveys, contextual inquiry, stakeholder interviews, A/B
testing,wireframes,sitemaps,personas,scenarios.Whatabouthuman
interaction,usability,andaccessibilitystudies?
Whatareyourdatasecurityandprivacyrequirements?
Howdopeoplesigninandbecomeauthorized?
Aretheremultiplestepsthatareprogressivelyrevealed,oneafter
another?
IsthereaneedforacustomizableUI?
Whatareyourglobalizationandlocalizationrequirements?
What devices are you targeting, such as desktop and mobile
platforms?
How can you keep data presentation to a minimum to not
overwhelmtheuser?
Whatformisdatabestpresentedin?
Whatisthebusinessneedthatcanbeaccomplished?
Whataccessibilityrequirementsarethere?
What are the navigation levels of the UI? Can you map out how
thenavigationworks?
Doyouneedauserfeedbackmechanism?
Doyouhaveofflinerequirements?
How is state stored in the client? Centralized across all view
hierarchy?
HowwilltheUIbedeployedandupdated?
Howwillusersenterdataandwhattestsareneededtovalidateit?
Can you map out the multi-step data entry forms and show the
branchingconditions?
Theanswerstothesequestionsshouldbecarefullydocumented.Beforeyou
roll anything out into your production environment, have experts reviewing
everything.
React is a great choice as a UI framework and fulfills all the needs of the
presentationlayeroftheapplicationarchitecture.ItusesJavaScript,soitfitsin
perfectly with the overall development stack we have been pursuing. You can
nowlearnthespecificsofReactandseeapracticalimplementationofhowitis
usedwiththeNewsWatchersampleapplication.
15.2IntroducingReact
ReactisaframeworkthatgivesyoutheabilitytodynamicallyrenderHTML
content in a browser web page. You write JavaScript to go alongside your
HTMLmarkup.TheJavaScriptcodecanleverageotheravailablelibrariestodo
things like HTTP/Rest requests, data binding and navigation and much more.
TheoverallcapabilitiesofReactprovidethemechanismstoenableyoutobuild
aSPA,atraditionalserver-siderenderedwebsite,orahybrid.
There have been other similar frameworks that have appeared at the same
timeasReactsuchasKnockout,js,Ember.js,Angular,Vue.jsandBackbone.js.
Reactismychoiceinthisbookasithasmanythingsgoingforit,suchasbeing
widelyadopted,andalsobecauseitisofficiallysponsoredbyalargecompany
(Facebook) as an open-source project. You can be up and running on the
desktop, mobile web, and even produce native mobile applications in a short
amountoftime.
Note:You see me referring to React as a framework.I do so in the larger
sense.React,alongwithyourchosenadditionsforotherfunctionality,providea
framework that you built your application presentation layer with. That is the
point of a framework, it is extensible. It just so happens that React is not as
prescriptive and feature capable as something like Angular. It is up to you to
combineitwiththeotherlibrariesofyourchoicetogiveyouallofthefeatures
youneed.
Reactfreesyoufromsomeofthelaboriouscodeyouusedtohavetowriteto
do DOM (Domain Object Model) manipulation. Because of its ability to bind
andaffectDOMelements,younolongerneedtoutilizelibrariessuchasjQuery.
ReactmakesHTMLdynamicsothatdatavaluesflowbackandforthforyou.
To use React, you write component files that render to the DOM. Be aware,
however,thatReactitselfdoesnotprovidecontrolelementsorstylingforyou,
asthatislefttotheHTMLmarkupandotherlibrariessuchasmaterialdesign,
Bootstrapandmanymorethatareavailable.
TheReactlibraryisconsumedbyplacingascriptinsideyourHTMLfilethat
pulls in React as JavaScript to execute from a CDN. You can also locally
bundledReactthroughWebpackandthenpullitinthatway.
Therearethird-partytoolsthatactasUIdesignstudios,whereyoucandrag
and drop elements and play around with them and have React code generated
andedited.Thisishandytobeabletovisualizeacomponentinanisolatedway
soastosaveyoutime.Otherwise,youneedtoseethecomponentinanoverall
application.LookintotoolssuchasStorybookandReactStyleguidist.
Note:Youmightrecallthatthe Node.jsExpress modulehasthe conceptof
serving up templates of “HTML-like” files and binding data to them on the
server-side,sothattheyarriveontheclientsideallfilledout.Expresssupports
many template formats such as Jade, EJS, mustache, and handlebars. I don’t
recommendthisserver-sidedatabindingtechnique.Instead,Igivepreferenceto
simplyservingtheHTMLthatworksasaSPAapplication,withtheclientside
usingReacttorequestdatathroughaRestAPIwithJSONandthendoingthe
binding on the client side. This alleviates the back-and-forth HTML page
requests. With a SPA, all of the navigation and page rendering is done on the
clientside,notontheserverside.Thisissimilartowhatyouwouldneedtodoif
youweretodevelopanativemobileapplication.YoucanalsostickwithHTML
insteadofhavingtolearnanewtemplatemarkupsyntaxsuchasJade.
15.3ReactwithonlyanHTMLfile
Youcould pullin the Reactlibrary witha scriptmarkup inan HTML file.
Thisisnotagoodidea,butIshowitasawaytounderstandwhatultimatelyhas
to happen to bring any code into an HTML page. React is just a library like
anythingelsethatrunsasascriptinabrowserthatexistsinanHTMLrendered
page.Youcanplacethistextinafileandopenitwithyourbrowserandseethat
itworks.
<!DOCTYPEhtml>
<html>
<head>
<metacharset="UTF-8"/>
<title>HelloWorld</title>
<script
src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.2/react.js">
</script>
<script
src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.2/react-dom.js">
</script>
<script
src="https://cdnjs.cloudflare.com/ajax/libs/babel-standalone/6.21.1/babel.min.js">
</script>
</head>
<body>
<divid="root"></div>
<scripttype="text/babel">
ReactDOM.render(
<h1>Hello,world!</h1>,
document.getElementById('root')
);
</script>
</body>
</html>
What you want to do, is install the necessary tooling on your developer
machinethatcanbeusedtodoabuildofyourcodetoproduceabundledfile
thathaseverythinginitfromyourprojecttouse.ThismeansReactisbundled
withyourwebsiteuploadofstaticfiles.YouendupwithageneratedHTMLfile
thatlookssomethinglikethefollowing:
<!DOCTYPEhtml>
<html>
<head>
<metacharset="UTF-8"/>
<title>HelloWorld</title>
</head>
<body>
<divid="root"></div>
<scripttype="text/javascript"
src="/static/js/main.35d639b7.js"></script>
</body>
</html>
The JavaScript file that you see being pulled in with the script tag is
generated in your build process on your machine and contains all the needed
ReactcodeandyourowncodethatisnecessarytoloadandrunyourSPApage.
Thenextsectionwillwalkyouthroughhowtoarriveatthefileshownabovein
thesimplestwaypossible.
15.4InstallationandAppCreation
If you really want to go through all of the work of installing each of the
modulesandtoolsfordoingreactdevelopmentyoucertainlycandothat.Todo
so, you can go to the official React website and follow their downloading
instructions.Youcanstartaprojectonyourownanddonpminstallsofmodules
such as react and react-dom. Then you would need to install tools for cross-
compilingandbundling.YouwouldwanttogettoolsinstalledfromNPMsuch
aswebpack,Babel,Autoprefixer,ESLint,Jest,andothertools.Youwouldneed
to become very familiar with the following terms and how to download and
managetoolsforeach.
cross-compiling:AllowsyoutousethelatestJavaScriptsyntaxandhaveit
turnedintoJavaScriptthatcanruninolderbrowsers.
bundling: This is how to gather all the files together for much better
performance and a controllable loading capability. Otherwise, you have to
reference the minified JavaScript files in your HTML and take the hit on the
initialpageload.Goingwithbundlingalsogetsyoupreparedforbeingableto
deploy as a native application on a mobile device, as that is the only way to
accomplishthat.
Insteadofinstalling thetoolingon yourown, youcansimply installatool
that populates a React project for you with some example code and all of the
tooling.
Usingcreate-react-app
The task of getting the initial project files in place to have the traditional
“Helloworld”applicationrunningmayseemalittledauntingatfirst.Thebest
approach is to install and use the create-react-app utility. Install this tool as
follows:
npminstall-gcreate-react-app
Youcouldnowrunthisandyouwouldhaveanapplicationallreadytorun.
Inourcase,wealreadyhaveanexistingfolderwithaNode.jsapplicationinit.
WhatwewilldoiscreatetheReactapplicationinaseparatefolder.Thensimply
copy over the necessary folders from the created project over into the Node
project. This is the best approach to take for now. Run the following on the
commandlineinacompletelynewdirectory.ThiswillcreateyourbasicReact
applicationthatyoucanpullfrom:
create-react-appmy-app
Copyoverthe“public”and“src”directories.Lookinthepackage.jsonand
the.gitignorefilesandbringoverwhatismissingintheonesintheNode.jsfiles.
Whileyouareeditingthe.gitignorefile,add“.chrome”asaline,asthiswillbe
neededlater.Forthepackage.jsonfilethatmeantjustbringingoverafewofthe
scriptcommands.Irenamedthem,soasnottoconflictwithanythingIhadfor
myNode.jsproject.
"start-react":"react-scriptsstart",
"build-react":"react-scriptsbuild",
"test-react":"react-scriptstest--env=jsdom"
You also need to copy the Node.js package.json lines that were for the
following dependencies: react, react-dom and react-scripts. Then you can run
npminstallandhavethosereadytouse.
To run the Reactapplication, you cannow run the command“react-scripts
start”andthatwouldbeservedupforyouonyourlocalmachine.Wehaveputa
script command in package.json for these. You can thus run “npm run start-
react”. If you run that, you will see the following launched in the Chrome
browser:
Figure90-StarterprojectscreenforReactinstall
Note:Node.jshasitsown commandstogetaninitial applicationstructure
set up and configured on your development machine. It can be confusing on
whattodo foryourinitialprojectcreation,asyou havejust seenthatthere is
alsoacommandlineutilitytocreateaReactapplication.Thereisalotmoreto
aReactinstall,however,especiallyifyouuseaframeworksetup,likethecreate-
react-apputility.ThusIhadtoessentiallygeteachprojectsetupindependently
andthenbringoneovertotheother.Thereareotherutilitiesouttheresuchas
Yeomanthatcangiveyouthecompletescaffoldingforeverythinginonesingle
initialization.Thereisalsoageneratornamedreact-fullstackthatcanbeusedas
astartingpointforyourproject.Ofcourse,youcanalsocloneanyone’sGitHub
application,suchasmine,andhaveeverythingallreadytouse.
ServingupReactfromNode.jsandExpress
Before getting into the heavy coding in React, make sure that the initial
React page is served up correctly from the Node.js service. The details of
making your Node.js Express application serve up the React index.html and
JavaScriptcodeasaWebPageareextremelysimple.Herearethebareminimum
linesofcodeyouneedtodothat.
constexpress=require('express');
constpath=require('path');
constapp=express();
app.get('/',function(req,res){
res.sendFile(path.join(__dirname,'build','index.html'));
});
app.use(express.static(path.join(__dirname,'build')));
app.listen(3000);
Inthenextchapter,youwillseethedetailsoftheNewsWatcherapplication
code for the presentation layer. You will see how these lines are incorporated
intotheserver.jsfile.TorunitwithNode,youputinthoselinesshownabove
andthenneedtofirstdoabuildandbundleoftheReactcodebeforetheNode
servicecanbestarteduptoserveit.TodothebuildandbundleofReact,yourun
thefollowing:
npmrunbuild-react
YoucouldthenpressF5inVisualStudioCodeandthenyourapplicationis
once again being served up, this time through Node, and you can go to
http://localhost:3000/ in your browser to see it. If you are curious, you might
wanttoinspectthebuilddirectorythatiscreatedandlookateachofthefilesto
seeexactlywhatisbeingservedandrunontheclientbrowser.
Theindex.htmltemplateandgeneratedone
If you recall, I showed some HTML that was all self-contained. It had
everythingneededtorunaReactbackedwebpage.Ithadthescriptsnecessary
tousethe Reactlibraries. However,withthe projectstructurecreated withthe
create-react-app utility, it works completely differently. You still have the
index.html file. That is used as the starting point for the one being generated.
Youwillnoticethatitdoesnothaveanyscriptspulledin.Itlooksasfollows:
<!doctypehtml>
<htmllang="en">
<head>
<metacharset="utf-8">
<metaname="viewport"content="width=device-width,initial-scale=1">
<linkrel="shortcuticon"href="%PUBLIC_URL%/favicon.ico">
<title>ReactApp</title>
</head>
<body>
<divid="root"></div>
</body>
</html>
ThisHTMLfileisservingasapointtemplatefileandhasthatsamedivwith
anidof“root”.Bytemplate,Imean,itistakenatbuildtimeandalteredandthe
finalindex.htmlfileiscreatedandplacedinthebuilddirectory.Hereiswhatthat
lookslike:
<!DOCTYPEhtml>
<htmllang="en">
<head>
<metacharset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,shrink-
to-fit=no">
<metaname="theme-color"content="#000000">
<linkrel="manifest"href="/manifest.json">
<linkrel="shortcuticon"href="/favicon.ico">
<title>ReactApp</title>
<linkhref="/static/css/main.c17080f1.css"rel="stylesheet">
</head>
<body>
<divid="root"></div>
<scripttype="text/javascript"
src="/static/js/main.35d639b7.js"></script>
</body>
</html>
Thestartingpointforyourcode
Wecanfinallygettotherealpointyouhavebeenwaitingfor.Youcannow
learnwhatthestartingpointisforwhereyouplaceyourowncustomcodefor
your React application. This code belongs in the index.js file found in the src
folder.ThisiswhatisusedatbuildtimetobethestartingpointofyourReact
app.HereisthefilethattheReactbuildtoolsknowaboutandusetobootstrap
yourapplication:
//index.js
importReactfrom'react';
importReactDOMfrom'react-dom';
importAppfrom'./App';
import'./index.css';
ReactDOM.render(
<App/>,
document.getElementById('root')
);
You see the same ReactDOM.render() call that was in the initial self-
containedHTMLfile.Itisjustsplitoutandusedatbuildtimetobepulledin.
Whatyouseehereisthattheindex.jsfileisimportingtheApp.jsfile.Inthere
youwillfindthedefinitionofthecomponentnamedAppthatisbeingrendered.
Reactcodeis basedontheconceptof componentsthatare assembledto build
theUI.Eventually,itallleadsdowntoHTML.
Theindex.jsfileisusedbyReacttodotheinitialelementinsertionintothe
divthathadthe“root”id.ThiscodeusestheApp.jsfileasthestartingpointfor
whereyouplacealloftheJavaScriptandHTMLtagstodotherenderingofwhat
yourUIwillbe.Hereiswhattheinitialcontentsarefortherenderingasfoundin
theApp.jsfilewhenwecreateditwiththecreate-react-appcommand:
//App.js
importReact,{Component}from'react';
importlogofrom'./logo.svg';
import'./App.css';
classAppextendsComponent{
render(){
return(
<divclassName="App">
<headerclassName="App-header">
<imgsrc={logo}className="App-logo"alt="logo"/>
<h1className="App-title">WelcometoReact</h1>
</header>
<pclassName="App-intro">
Togetstarted,edit<code>src/App.js</code>andsavetoreload.
</p>
</div>
);
}
}
exportdefaultApp;
Whenyourun“npmrunbuild-react”fromthecommandline,thisbuildsand
bundleseverythingforusagetobe servedupfromyour Node.jsservice.After
thebuildcommandcompletes,youwillfindthatyouhaveanewfoldernamed
“build”.Itisinherethatthefilesareplacedthatarefullycapableofbeingused
aspagestobeloadedandseeninabrowser.Forexample,youwillnowfinda
newindex.htmlfileinthebuildfolderthatwasshownalready.
Thebigdifferencenowbetweenthisfileandtheoneinyourpublicfolderis
there is a style sheet and script file brought in that are used to run the React
application. This is why a Node.js server can be set up to serve up the build
folder index.html file and also how it can get at its own built files for all the
ReactandprovidedJavaScriptfilestorun.
DebuggingReactcode
Youwillwant to know how to debugthis new React code. Debugging the
Node.jscodewassimple.YoujustplacedabreakpointinyourNodeJavaScript
codeandyoustoponitwhenthatisexecutedandstepthroughit.Section18.3
willcontainmoreonthistopic.
ForReactcode,itisreallyeasytodebugyourcodeintheChromebrowser.
YoucangettothecodeandsetbreakpointsbyselectingfromtheChromemenu
More tools->Developertools. You can then be in an environment where you
canalsoinspecttheDOMandlookatconsoleoutputandmuchmore.
You will need to run Node separately now, either from the command line
(“npm start”), or in VS Code you can press F5. Then you can go to
http://localhost:3000/inChromeandwiththedevelopertools,debugtheReact
code.Youcan,atthesametime,debugserver-sideNodecodeinVSCodeasitis
running.
YoucanlookuponlinetofindouthowtousetheChromeDevelopertools.
Tosetabreakpoint,youjustopentheSourcestabandthenontheleftsideyou
willfindthefolderswiththecodeinit.Gotothestaticfolderandinthere,will
bethejsfolder.Opentheviewsfolderandselectafilesuchasloginview.jsand
youcansetbreakpointsandstepthroughthecode.
Thenicethingthatishappening,isthatyouarelookingatyourcodebefore
itwascross-compiledandbundled.Therearemappingfilescreatedbehindthe
scenesthatallowtheactualcodethatisbeingruntobemappedtotheoriginal
lines for your ease in debugging it all. Here is what it looks like with
NewWatcherrunningandhittingabreakpointinthecode.
Figure91-ChromeDevelopertools
15.5TheBasicsofReactrenderingwith
Components
PerhapsyouhaveheardthatReactdoesnotuseHTMLfileslikeatraditional
WebServermightserveup.Reactinsteadusescodefilestocreatecomponents
thatgetpiecedtogethertocreateyourUI.ItisalldonethroughstandardDOM
manipulationcode atthe lowest level.Don’tbe thinkingyou are gettingaway
from HTML markup and CSS styling by any means. There will ultimately be
justasmuchofthatrequiredtocreateyourapplication.
WithalltheReactcomponentabstractionsyoupiecetogether,atthelowest
levelyouwillendupprovidingHTMLmarkupthatgetsrenderedtotheDOM.
YouhaveallthestandardsetofHTMLelementstopullfrom.Youwillusetags
suchas<p>,<a>,<ul>,<button>,<input>,<iframe>,<img>etc.Youcanalso
addcontent,attributesandstyling.
ReacthasyouprovideHTML,butyoudoitinaJavaScriptfile(.jsor.jsx)
instead.ThecodeisJavaScriptthatallowscertainadditionalsyntax.Thesefiles
arecross-compiledintotheactualJavaScriptcodethataBrowsercanrun.
Atthe startof thischapter,you learnedthat thereis a top-levelindex.html
filethathasadivelementinitasfollows(seethelineinboldtype):
<body>
<divid="root"></div>
<scripttype="text/javascript"src="/static/js/main.3dd1fbc9.js"></script>
</body>
IfyouunderstandalittleaboutbrowserDOMmanipulationAPIs,youknow
thatyoucancreateanythingyouwantintheDOMthroughcode.Forexample,
you might insert a text element using JavaScript code. Furthermore, you can
referenceanyexistingelementbyanIDyougiveit.Inthatway,youcanfindit
in code and alter it. This means you could find a div and add a child HTML
elementtoit.ThatchildelementwouldthenshowupintherenderedUIaswell.
Hereis some plain HTML code with no React usage at all. Youcan place
thisin afile and openit in yourbrowser totry it out.It uses someJavaScript
code inside a script tag. The standard document object available in browser
JavaScript is used to access your web page. It is part of an agreed-upon API
standardavailableinanybrowser.Thecodecreatesatextelementasachildof
thedivelement.
<!DOCTYPEhtml>
<body>
<divid="root"></div>
</body>
<script>
vartextnode=document.createTextNode("WelcometoReact");
document.getElementById("root").appendChild(textnode);
</script>
</html>
React employs this exact mechanism to take anything you want to be
rendered and presented on the page. To begin with, React is rendering to the
maindivyousawabovewiththeidof“root”.Thereisastartingpieceofcodein
afilenamedindex.jsthatisusedasthepagegetsrendered.Atthetimetheinitial
Reactapplicationwascreated,itlookedasfollows:
//index.js
importReactfrom'react';
importReactDOMfrom'react-dom';
importAppfrom'./App';
import'./index.css';
ReactDOM.render(<App/>,document.getElementById('root'));
ThiscodeeventuallygetsbuiltandmorphedintoanotherJavaScriptfilethat
utilizestheReactAPItomakecallsthatusethedocumentobjectanddothings
likecallappendChild.ThisisthefirstpieceofcodethattheReactlibraryusesto
rendersomething.
ThecallyouseetoReactDOM.render(…)isacceptingasthefirstargument
whatelementyouwanttorender.Appisacomponentfromanotherfilethatwas
imported. The second argument to the render() call is telling React where to
rendertheUI.Thatiswherewearereferringtothe“root”divintheindex.html
file.
ThelastfiletolookatintheprocessofunderstandinghowReactdoesDOM
rendering,isthefilethatcontainsourcustomdefinedcomponent.Inthiscase,
thefileisnamedApp.js.Thisthenisthepatternforthecodethatyouwillcreate
over and over as you define the UI you want to be rendered. Here are the
contentsofthatfile:
//App.js
importReact,{Component}from'react';
importlogofrom'./logo.svg';
import'./App.css';
classAppextendsComponent{
render(){
return(
<divclassName="App">
<divclassName="App-header">
<imgsrc={logo}className="App-logo"alt="logo"/>
<h2>WelcometoReact</h2>
</div>
<p>edit<code>src/App.js</code>toreload.</p>
</div>
);
}
}
exportdefaultApp;
ReactisaJavaScriptlibrarythatexposesanAPI.PartofthatAPIisaclass
youcanderivefromthatitunderstands.Componentisaclassyouderivefrom
and write code to contain your own UI to render. App is your highest parent
controlandisderivedfromComponent.Youwillcreateacompletehierarchyof
othercomponentsyoudefineandmakeuseof.Intheapp.jsfile,youexportthe
Appcomponentsothatitcanbeimportedandusedintheindex.jsfile.
Noticethefirstlineofapp.jsthatdoesanimportofReact.Thismustpresent
even though you don’t see the usage of that object. This is because the
JavaScriptcodeistransformedbyBabel,andwillbeusedinthatcode.
Note:Byconvention,youstartyourcustomcomponentswithanuppercase
letter.ThisthenhelpsyoudistinguishthesefromHTMLelementsthatarealways
alllowercase(i.e.div).
TherenderMethod
The Component class requires you to provide a method named “render”.
ReacttakesyourAppcomponentandcallstherender()methodtogetanything
thatneedstoberendered.BeawarethatthisisnotactuallyaJavaScriptfilethat
canberun.Asexplained,itisusedintheReactbuildprocessandturneditinto
something that can be run. You may have seen that the return in the render
methodisnotreallyreturninganythingacceptableinJavaScript.
ThisuseofJavaScriptiswhatisreferredtoastheJSXsyntax.Itallowsyou
tomixHTMLtagsandJavaScriptrightinonefile.Forexample,youcanhave
conditionalcode that determineswhat elementsshould be rendered on apage.
Lookatthefollowingexample:
render(){
if(user.account.total>1000){
return(<h1>Youmaintainedtheminimumrequiredbalance!</h1>);
}
return(<h1>Youarebelowyourminimumrequiredbalance</h1>);
}
BeawarethatyoucanonlyreturnonesingleHTMLelementfromtherender
function.Meaning,thereneedstobeasingleparentelementsuchasadiv.For
example, you can’t return two h1 elements unless they are wrapped in a div
element.Youcanreturnasingleh1elementasshown.
There are a few subtle differences in the HTML as it is used with React.
Theseareasfollows:
The React DOM property naming convention is to use camel
casing and has altered some of the standard properties slightly. For
example,tabindex,becomestabIndexandclassbecomesclassName.
You use curly braces to embed JavaScript expressions in an
attribute. For example, specifying the source image for an <img>
element,youwouldhavetodothis:src={myObject.someImageURL}.
Youcanuseanyobjectinyourcodethisway.
To summarize, you can understand that a React application consists of
providedComponentsthatcontainHTMLtoberendered.Thisthenistakenby
theReactAPI,andusingthedocumentobject,Reactrendersthatintotheactual
browserDOM.
Note: The React API also has calls for creating individual elements. For
example, there is a method React.createElement() that takes arguments
specifyingtheelementtag,attributesandcontent.ThisisactuallywhattheJSX
getsturnedintointhebuildprocesswheretheBabelcompilerisrun.Thisbook
willstickwithusingComponentsinJSXfilesyntax.
MoreaboutmixingJavaScriptwithmarkup
YoubrieflysawthatinJSXsyntax,youcanhaveconditionalcodethatuses
controlflowofJavaScripttodecidewhatmarkuptoreturnforrendering.Thisis
reallyoneofthegreatadvantagesofusingReact.
Imagineyouhadsomebackendservicethatyoucalltoretrievesomedata,
suchasfromanHTTP/Restwebservice.Youcantakethatdataandthenprocess
itintheJSXcodeandproducewhateverUIyoulike.
Thenext examplecode willshow the takingof some datathat showshow
codecanmakeuseofit.Thedataisalistofcommentstobedisplayed.Thecode
will first determine an appropriate piece of text to display and will then loop
through the comments and display each using another lower level component.
TheloopingcodemakesuseoftheJavaScriptmap()functionofthearrayobject
toloopthroughandgeneratetheHTML.
The CommentListItem is some custom component you would provide that
wouldbereusableandknowhowtorenderanindividualcomment.
classCommentsextendsComponent{
render(){
constcoms=props.comments;
return(
{coms.length>0&&
<h2>Thereare{coms.length}comments</h2>
}
<ul>
{coms.map((comment,idx)=>
<CommentListItemkey={idx}
title={comment.title}/>
)}
</ul>
);
}
}
Note:YouhavethefullpowerofpullinginJavaScriptlibrarieslikelodash
thatcanbeusedtoproducelogictooutputthemarkupyouwantrendered.
You can see what is possible in code that combines JavaScript code with
HTMLtagsbeingreturned.JavaScriptcanbeinsertedinthemiddleofHTML
markupbyplacingitinsidecurlybraces.
Youcansetupblocksofcodethatrenderbasedonatest,suchastheoneyou
seetestingcoms.length.Finally,youseetheuseofthemap()functionthatlets
youloopthroughanditeratetorenderacomponent.Inthiscase,itisutilizing
anothercomponenttocreatethelielementsthat wouldbeplacedinsidetheul
elementtocontaineachlistitem.
Note:IfyoueverhavelogicthatneedstodecidebetweenshowingsomeUI
orhidingit,itisbettertonotuseCSStohidethetags,butinsteadtojustnotto
returnanythingatall.Yousimplyreturnnullfromthecodeandthatisit.
Alternatesyntaxusingafunction
This book will always show components being built that are derived from
Component. Theres is an alternate syntax that can be used that you should be
awareofincaseyoueverseeit.
Here is a component that is shown using the Component class syntax first
and then is shown using the alternate syntax. One is using the arrow function
syntax.Allareconsumedinthesamewayandhavethepropsavailabletopass
in.Withtheclassusage,thepropsareofcoursepartoftheclassinstance.
//CodeusingaComponentclass
classGreeting1extendsReact.Component{
render(){
return<h1>Greetings,{this.props.name}</h1>;
}
}
//Codeasafunction
functionGreeting2(props){
return<h1>Greetings,{props.name}</h1>;
}
//Witharrowfunctionsyntax
constGreeting3=({name})=>{
<h1>Greetings,{name}</h1>;
}
//Usage
constg1=<Greeting1name="Joe"/>;
constg2=<Greeting2name="Mary"/>;
constg3=<Greeting3name="Paul"/>;
Therearelimitationsifyouusethefunctionsyntax.Forexample,youcannot
use Redux. You will also learn about what lifecycle events are, and find that
theseareonlyavailableifyouusetheComponentclass.Youwillonlyseecode
examplesinthisbookusingtheReact.Componentclassextension.
15.6CustomComponentsandProps
YouhaveseenasimplecomponentnamedApp.Thiswasusedintheinitially
createdReactapplication.ItderivedfromtheReactComponentclassandhada
render()method.Anyuser-definedcomponentcanactasawrapperaroundthe
complexity of some underlying UI logic. Each component is really existing in
ordertorendersomeHTML.Youcreatecomponentstosplitfunctionalityoutin
awaythatcanbeself-containedandreusable.
Componentscanbeverycomplexandcanhavepropertiespassedintothem
tocontroldifferentaspectsoftheirrendering.Forexample,youcouldpassinan
arrayofdatathatthecomponentthenrenders.
Componentscanmaintainstate,suchasdataaboutwhatisgoingonwiththe
component. Passed in properties and internal state can be updated
asynchronously and the UI gets rendered when the changes happen. React
detectsthechanges,andupdateswillbebatchedtoprovidebetterperformance.
The capability of having properties on a component is what React calls
“props”.Acomponentcanhavemultiplepropsonit.Theirusagelooksjustlike
thatofattributeswhenyouuseacomponentinothermarkup.Forexample,you
couldpass insome textas aprop asshown inthe followingexample. We can
altertheindex.jsfiletolookasfollows:
ReactDOM.render(
<Appname="Joe"/>,
document.getElementById('root')
);
YoucanaltertheAppcomponentasfollowsandthenseeitwork.Hereisthe
altered App.js file to use the passed in name. The text is taken and displayed
alongwiththewelcometextthatspecifiesaperson’sname.Thelineinboldis
theonlymodificationneeded.
//App.js
importReact,{Component}from'react';
importlogofrom'./logo.svg';
import'./App.css';
classAppextendsComponent{
render(){
return(
<divclassName="App">
<divclassName="App-header">
<imgsrc={logo}className="App-logo"alt="logo"/>
<h2>WelcometoReact{this.props.name}</h2>
</div>
<pclassName="App-intro">
Togetstarted,edit<code>src/App.js</code>andsavetoreload.
</p>
</div>
);
}
}
exportdefaultApp;
Everypropshowsupinthecomponentaspropertiesonthepropsobject,and
are available in the component class via this.props. The name property is
accessedintheprecedingexampleasthis.props.name.
A property is only used as an input mechanism to a component. It is
importanttoknowthatPropsareimmutableinsidethecomponent.Forexample,
you could not have code that alters this.props.name. Props are meant to be
passed in and consumed by the Component when it renders itself. The calling
codeiswhatsetsthepropsandcanchangethemifneeded.Ifthecallingcode
were to have a timer that later changed the name prop, then the UI would re-
render.
Youwilllearnthattherearewaysintheparentusageofacomponentwhere
itcanbenotifiedofsomechangeofdataacquiredbyachildcomponent.Thisis
donebypassingacallbackfunctionasapropthatthechildcomponentcanuse.
This is totally up to you to define. There are predefined callbacks that can be
used as standard HTML attribute notification mechanisms, such as OnClick().
These,however,arenotsurfacedbackuptotheparent,unlessyoutakestepsto
providethemechanismtofeeditbackup.
15.7ComponentsandState
Aswasmentioned,propertiesarepassedintoacomponenttobeconsumed
via this.props. The other object available is this.state. State is internal to the
componentandcanbechangedinternally.Reactknowsaboutstatechangesand
canrefreshthe UItoreflect anychanges.Look atthefollowing codeand you
willfindtheusethethis.stateobject.Hereisalsowhereyouwillseetheusageof
aconstructorforyourcomponentclass.
classAppextendsComponent{
constructor(props){
super(props);
this.state={name:this.props.name};
}
render(){
setTimeout(()=>{this.setState(currState=>({
name:"John"
}))},5000);
return(
<div>
<h2>WelcometoReact{this.state.name}</h2>
</div>
);
}
}
It is only in the constructor that you can make any direct assignment to
this.state.Inotherplacesinyourcode,youcallthis.setState()tomakechanges.
Also note that you call super(), which is the Component base class call for
initializingitsconstructor.Youcanseeinthecodethattheinitialnameproperty
issetfromthepropsthatarepassedinandthenalteredinatimercallback.
Anytime you alter the state outside of the constructor, you must use the
this.setState().You only need to provide the properties you want to refresh in
each case. It won’t affect any of the other state properties of the component.
Reactwilldo a UIrefresh whenit sees statechanges havehappened andmay
batchupseveralbeforeitdoesaUIrefresh.
Withtheusageofthis.setState()youcanaccessthecurrentstatebeforeyou
makeyourchange.Inthepreviousexample,wedon’tactuallyusethatcapability
intheexample.
The state should be moved up to the highest component which has it in
commonacrosschildcomponents.Aparentcanhaveastatepropertyofitsown
andpassthatintothechildcomponentasaprop.Whentheparentstatechanges,
thechildwillseetheupdateandReactwillre-renderthecomponentifnecessary.
Model,ViewandController
IfyouwanttothinkintermsofhowanMVCarchitectureworks, youcan
recognize that React fits into this model by providing both the View and the
Controller capability. The View is what renders DOM elements from your
Component render method. The Controller is the rest of the logic that acts on
behalfofthepassedinpropsandmanagedstate.
YourReactcomponentcandothefetchingoftheModeldata.Youhavecode
fetchthedataandpassitaspropstoacomponent,orinthecomponentitselfyou
fetchitandsetthestate.YoucanutilizelibrariessuchasFlux/Reduxtodothe
coordination of data fetching and the setting of state in the code of the
component. Here is an image that might help you visualize how this all fits
togetherinanMVCdesign:
Figure92-ReactMVCconcepts
Note:Youcandebateexactlywheretokeepthecontrollerandtheviewcode.
Youcouldhaveitinonesinglecomponent.Itmightbebest,however,toseparate
the controller code from the view code. To do this you could have a data
componentthatusesadatastoreandthenthedatacomponentwouldpassprops
to a view only presentation component. These two types of components are
called Container and Presentation Components (also called "smart" and
"dumb").MorewillbesaidaboutthislaterinthesectiononReduxusage.
15.8EventHandlers
TheHTMLtagsusedinrenderingcanhaveeventhandlersgiventothemas
appropriate.Thiswouldbeforeventssuchasaclickorkeypress.InReactJSX
files, this means you provide a JavaScript function in your React Component
class and then use curly braces for specifying the event handler. Events are
specifiedwithcamelcasecharacters.
Hereisanexamplethataddsahandlerforaclickeventonabutton.Notice
thearrowfunctionisnecessarytohavethecorrectcontextfor“this”.Otherwise,
thethis.setState()wouldhaveanerrorthrown.
classAppextendsComponent{
constructor(props){
super(props);
this.state={count:0};
}
handleClick=(e)=>{
this.setState(currState=>({count:currState.count+1}));
}
render(){
return(
<div>
<h1>Youhaveclickedthebutton{this.state.count}times</h1>
<buttononClick={this.handleClick}>
Clickit
</button>
</div>
);
}
}
The “e” function argument is actually an event object that has a lot of
propertiesandmethodsonittobeusedasneeded.Forexample,youmightwant
tocaptureacertainkeythatwaspressedandthenpreventitfrompropagating.
Therearecaseswhereyoudon’twantaclick,orkeypresstobeprocessedany
further.Topreventpropagationofanevent,youwouldcalle.preventDefault().
PassingupEventHandling
Youcanhavetheconsumerofacustomcomponentbenotifiedofeventsina
sub-componentbyexposingthemasproperties.Ifyouhavemultipleevents,you
needtonameeacheventpropsomethingdifferent.Inthecodebelow,MyButton
just calls the prop callback method in its onClick event. In this way, the App
componentcanhavethecodethatrespondstothebuttonclick.
classAppextendsComponent{
constructor(props){
super(props);
this.state={count:0};
//Alternatewaytoensurecorrectthisbinding
this.handleClick=this.handleClick.bind(this);
}
handleClick(e){
this.setState(currState=>({count:currState.count+1}));
}
render(){
return(
<div>
<h1>Youhaveclickedthebutton{this.state.count}times</h1>
<MyButtonmyClickHandler={this.handleClick}/>
</div>
);
}
}
classMyButtonextendsComponent{
render(){
return(
<div>
<buttononClick={this.props.myClickHandler}>
Clickit
</button>
</div>
);
}
}
The code above also shows the alternate syntax for binding to have the
correctcallingonthecallback,otherwise,youcanusethearrowfunctionsyntax.
15.9ComponentContainment
There is a special props property that is always available on a component
thatisused forpassing inchildcomponentsorchild HTMLelements directly.
Thenameofthepropertyis“children”.Thismeansthatyoucanenclosechildren
intheusageofacustomcomponenttag,andthengetaccesstothem.Hereisthe
previous example, but this time, the text of the button is passed as part of the
children.
classAppextendsComponent{
constructor(props){
super(props);
this.state={count:0};
}
handleClick=(e)=>{
this.setState(currState=>({
count:currState.count+1
}));
}
render(){
return(
<div>
<h1>Youhaveclickedthebutton{this.state.count}times</h1>
<MyButtonmyClickHandler={this.handleClick}>
<h1>Clickit</h1>
</MyButton>
</div>
);
}
}
classMyButtonextendsComponent{
render(){
return(
<div>
<buttononClick={this.props.myClickHandler}>
{this.props.children}
</button>
</div>
);
}
}
The previous example shows an HTML tag being enclosed as part of the
children.YoucanalsohaveotherJSXcomponentsorstringliteralsbechildren
aswell.
Note:WhentheJSXistakenandrendered,anywhitespaceatthebeginning
andendofalineisremoved,aswellasblanklines.Newlinesthatoccurinthe
middleofstringliteralsarecondenseddowntoasinglespacecharacter.
You can always pass in any name for a property that contains multiple
elements. This may become necessary if you have multiple sub-components
being used in elements and are displaying them. For example, in the previous
examplethebuttontextcouldhavebeenpassedinasfollows:
classMyButtonextendsComponent{
render(){
return(
<div>
<buttononClick={this.props.myClickHandler}>
{this.props.myButtonText}
</button>
</div>
);
}
}
15.10HTMLForms
Reactsupports the creation of HTML forms via the standard formtag that
existsinHTML.Youcancreateacomponentthatrendersitselfandprovidesthe
formtagandsupportstheeventhandlingfunctionforthesubmission.Asimple
exampleisasfollow:
classAppextendsReact.Component{
constructor(props){
super(props);
this.state={value:''};
}
handleChange=(event)=>{
this.setState({value:event.target.value});
}
handleSubmit=(event)=>{
alert('Thetestsubmittedis:'+this.state.value);
event.preventDefault();
}
render(){
return(
<formonSubmit={this.handleSubmit}>
<h2>Textis{this.state.value}</h2>
<label>
Name:
<inputtype="text"onChange={this.handleChange}/>
</label>
<inputtype="submit"value="Submit"/>
</form>
);
}
}
Notonlydoestheabovecodehavethesubmissionhandling,butyoualsosee
codetohandlethebindingforchangesofaninputelementsothatthechanges
are rendered. You can also get events on changes for a checkbox, list, select
controlandothers.TheonChangepropertyisusedforthoseevents.
15.11LifecycleofaComponent
React sets up a series of stages that a component goes through as it gets
renderedandeventuallydestroyedandtakenoutoftheDOM.Youcanprovide
methodsonyourclassthatgetcalledwhenthecomponententerseachofthese
stages.
These methods are called “lifecycle hooks”. For example, the
componentDidMount() method is called after the component output has been
rendered to the DOM. You can then know that your component is visible and
take any action you need. You might want to create a timer that refreshes the
state of the component every few seconds, if data changes in the backend
serviceslayermighthavehappened.
AnothermethodavailableiscomponentWillUnmount(),whichiscalledjust
beforeacomponentistakenoutoftheDOM.
15.12TypecheckingyourProps
Youcanmakeuse ofthepropTypespropertyofyourcomponentto setthe
expected property types of the usage of your component. This way, you can
catcherrorsinitsusage.Thisisonlydonewhenrunningindevelopmentmode.
You can test for individual props such as an object, array, func, string,
numberandmanyothertypes.Youcanalsospecifyifagivenpropisrequiredor
not. There is also a means of specifying default values for props that are not
passedinbyaparent.Hereisanexamplethatshowsoffthesecapabilities:
importPropTypesfrom'prop-types';
classAppextendsComponent{
render(){
return(
<divclassName="App">
<divclassName="App-header">
<h2>WelcometoReact{this.props.name1}</h2>
<h2>WelcometoReact{this.props.name2}</h2>
</div>
</div>
);
}
}
App.propTypes={
name1:PropTypes.string.isRequired,
name2:PropTypes.string
};
App.defaultProps={
name2:'Unknownperson'
};
15.13GettingareferencetoaDOM
element
There may be cases where you need to call the actual DOM manipulation
functionsonanelement.Todothat,youneedtogetareferencetobeabletouse
it. React has a special syntax that you can employ to set the reference to the
element.Hereiswhatthatlookslike.Youwoulddothisifyoucan’tstoreaway
areferenceinyourcomponentclasstoaninputelement.
<inputtype="text"ref={(input)=>{this.textInput=input;}}/>
Insomeothercode,youcouldmakeacalltousethis.textInputtogetorset
its value. There is also a function provided in the React library named
findDOMNode that can be used to find any element in code. I prefer to find
waysaroundusingeitherofthesemechanismsastheyreallyaretherejustforthe
oddcase,asthereusuallyisawaytofindanycomponentincontext,andgetits
valueinsimplerways.
Chapter16:FurtherTopics
ThataboutcoversthebasicsofusingReact.Thischapterwillcoverfurther
topicsthatyouwillwanttoknowtoachieveafullrobustapplication.
16.1UsingReactRouter
Itisassumedthatawebsitewillhavemanydifferentpagestobeviewed.Ina
SPA,thismeansmanydifferentviewswillberenderedasneeded.Toaccomplish
thisyouwillneedtorendersometypeofnavigationbarwithlinkstoclickon.
Fromthoselinkstheviewwouldbechanged.
React does not come with any built-in capability for handling page view
routing. You could certainly create your own code to render some type of
navigation bar across the top of your main site page and then do the work to
changetodifferentpageviewsinsideofthat.
ThesolutionIwillgiveinthisbookistoutilizeapopularopensourcenpm
packagenamedReactRouter.Thereareactuallytwodifferentpackagesyoucan
findthatarecreatedbythesamegroup ofpeople.Oneisforwebsites andthe
otherisfornativeappdevelopment.Wewillusethewebsiteversion,whichis
foundhere-https://www.npmjs.com/package/react-router-dom.Incaseyouare
interested, the native app version is named react-router-native. React Router
givesyoutheabilitytocreateadynamicnavigationexperienceonyoursite.To
getstarted,youcaninstallthepackageintoyourprojectasfollows:
npminstall--savereact-router-dom
Note:ReactRoutercanbeutilizedontheserver-sidetoreturnpagesthatare
renderedwithdata,orReactRoutercanbeusedapureclient-sideSPAsite.This
bookwillfocusontheclient-sideSPAusageandshowhowviewsarepopulated
fromclient-sidefetchesofdatafromtheserverHTTP/RestAPIcreatedinpart
twoofthisbook.
YoucanputatthetopofyourApp.jsfileanimportstatementtopullinthe
needed components that are provided by the react-router-dom package. These
wouldbeHashRouter,RouteandLink.Hereisthecodethatcouldbeplacedinto
App.jsforasimpleusagethathasnoCSSstylingforit:
//App.js
importReact,{Component}from'react';
import'./App.css';
import{HashRouter,Route,Link}from'react-router-dom'
constHome=()=>(
<div>
<h2>Home</h2>
</div>
)
constAbout=()=>(
<div>
<h2>About</h2>
</div>
)
classAppextendsReact.Component{
render(){
return(
<HashRouter>
<div>
<Linkto="/"replace>Home</Link>
<Linkto="/about"replace>About</Link>
<hr/>
<Routepath="/about"component={About}/>
<Routeexactpath="/"component={Home}/>
<Routecomponent={NotFound}/>
</div>
</HashRouter>
);
}
}
exportdefaultApp;
TheHashRouter is the main component wrapper that creates the container
for the UI to be rendered as navigation happens. Different components are
renderedasassignedbythe“component”attributeoftheRoute.Forasitethat
youwantbrowserhistorykept,useBrowserRouterinsteadofHashRouter.Inthe
above case, we are discussing a SPA, which would be implemented with the
hashrouting.
Theattribute“exact”isusedtomatchonlyasinglespecificpath.Ifyouhave
multiple components being rendered at a time, this is needed and will fix that
problem.
TheLinkcomponentisusedasananchortagmightalsobeused,butadds
theprop“to”forspecifyingtheroutetotransitionto.TheLinkcomponentworks
hand-in-hand with the Route component which specifies what component to
render for a given route. The “replace” attribute of the Link specifies that
clicking the link will replace the current entry in the history stack instead of
addinganewone.ThisiswhatwewantforaSPAclient-sidesite.
ThelastRoutelistedisonethatletsallotherpathsberenderedwithanerror.
Ifyouendupthere,youwouldprovideaNotFoundcomponenttostatethatthis
isa“404”,becauseitwasnotexpected.Forexample,thisNotFoundUIwould
renderifyouwenttosomethinglikelocalhost:3000/#/blahblahblah.
HereiswhattheUIendsuplookinglikeforthispreviousexamplecode:
Figure93-ReactRouterusage
16.2UsingBootstrapwithReact
Stylingawebsitetocreateamodernresponsivesitethatlooksgoodinafull-
screenbrowseraswellasonasmallmobilephoneisachallenge.Thereismuch
toconsiderwhenconstructingyourHTMLinthiscase.Thisbookisnotaway
foryoutolearnCSSorresponsiveWebDesigntechniques.Whatthisbookwill
doinsteadistakeasimpleandeffectiveapproachbyusingBootstrap.
Bootstrap was created by Twitter and is a set of UI HTML templates and
CSSstylesthatyoucanusetoaccomplishresponsivewebsitedesigns.
TomakethiseveneasierinaReactapplication,thereisannpmpackagethat
exportsReactComponentstobeeasilyused.Youjustneedtorunthefollowing
installtogeteverythingavailableintheproject:
npminstall--savereact-bootstrapbootstrap@3
TheonlyotherchangetomakeistopullintheCSSfilesneededsothatthey
areexposedforusageintheapplication.Hereisthealteredindex.jsfilethatis
foundinthesrcdirectory.Theaddedlinesareinbold.
//index.js
importReactfrom'react';
importReactDOMfrom'react-dom';
importAppfrom'./App';
import'bootstrap/dist/css/bootstrap.css';
import'bootstrap/dist/css/bootstrap-theme.css';
import'./index.css';
ReactDOM.render(
<App/>,
document.getElementById('root')
);
HereisanalteredApp.jsfilethatnowshowstheusageofaBootstrapstyled
button:
//App.js
importReact,{Component}from'react';
import{Button}from'react-bootstrap';
classAppextendsComponent{
render(){
return(
<div>
<h1>WelcometoReactwithBootstrap</h1>
<p><Button
bsStyle="success"
bsSize="large"
href="http://react-bootstrap.github.io/components.html"
target="_blank">
ViewReactBootstrapDocs
</Button></p>
</div>
);
}
}
exportdefaultApp;
Hereiswhattherenderingoftheexampleabovewouldlooklike:
Figure94-BootstrapusagewithReact
You can visit this React-Bootstrap site (https://react-
bootstrap.github.io/components/alerts/) to see all of the available components
youcanuseinyourReactapplication.TheNewsWatcherapplicationwillmake
useofseveralofthem.
16.3MakingHTTP/RestRequests
EveryapplicationthatyoucreatewillmostlikelyneedtomakeHTTP/Rest
requeststosomeback-endservicetoretrievedata.Reactdoesnothaveanysuch
capability built in to do that. This was done on purpose, as there are several
librariesthatarealreadygoodatthis.Oneoptionistousethe,builtinstandard
of fetch() as found in the JavaScript standard. Other npm modules such as
superagentcanbeinstalledandused.Thisbookwillutilizefetch().
Typically,youwillneedtofetchdatainthecomponentDidMount()method.
That code will update some internal state properties and in turn would cause
React to do any DOM updates necessary. Here is an example that outputs the
fetcheddatatotheconsolelog:
importsuperagentfrom'superagent';
...Codeleftout...
componentDidMount(){
fetch(`/api/blah`,{
method:'GET',
headers:newHeaders({
'x-auth':this.props.session.token
})
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
console.log(response.json);
})
.catch(error=>{
console.log(`GETfailed:${error.message}`);
});
}
This type of code will be used in when NewsWatcher is put together. The
basicusageis thatyouspecify theURIor relativepath.Aspart ofthe second
parameterobject,yougivetheHTTPverbusage,suchasput,post,getordelete.
Youcanalsosetanyheadersyouneed.Fetchcanbeusedaspromisebasedcode,
and that is what is shown here. You can verify the HTTP code returned and
handleerrorsandgetatthebodyofthereturneddata.
16.4StatemanagementwithRedux
Youhavelearnedhowtousepropsandstate.Stateandsometimespropsthat
aresharedshouldnotbekeptaspartofacomponentclass,butinsteadshouldbe
keptinacentralstoragelocationforallcomponentstogetat.Ifyoufindthatthe
propsorstateofacomponentreallyneedstobesharedwithothercomponents,
theyshouldbestoredcentrally.
Redux is a library that helps you share state across all your React
components.Itspurposeistoprovideacentralrepositoryforallcomponentsto
writetoandreceiveupdatesfrom.Thinkofitasfulfillingthesamefunctionthat
state does in a component, but across all components in a shared way. With
Redux,youcanalsoaccomplishdatatransformationsandroutingintheprocess.
Thestatethatyoustorecanbesomedatayoucollectedfromuserinteraction,
ordatathatcomesfromanetworkrequestandmuchmore.Justaboutanytype
ofsimpleJavaScriptdataaspartofanobjectcanbekeptthere.
Imagine retrieving data that needs to be shared across the hierarchy of
components.Forexample,theUIinonepartofthecomponenthierarchymight
havestatethatneedstobesharedwithanynumberofothercomponentsstrewn
alloverinthehierarchy.Itwouldgetconfusingifyoutriedtosendthatstateup,
across and down to another component. That would quickly become
incomprehensible.
Reduxsolves these problems by being the central state repository for your
completehierarchyacrossallcomponents.TouseRedux,youfirstneedtocreate
the Redux store. You do this in your code where it is initialized. When you
createthestore,youpassintoitthereducers.Areduceriswhatgetsachanceto
determinethestatechangesyouwanttomakewhendataflowsintheformofan
action. This reducer could be a single function, or a combination of reducers.
Thiswillbecomeclearerasyoulookatsomecodeexamples.
Thenpm packagethatyou installis called “redux”.The Redux objectthat
you can access has only four methods on it, so it is fairly easy to understand.
There are then just a few nuances to understand about each of the four calls.
Here is the creation of a store and also some code showing the four Redux
functions:
import{createStore}from'redux'
conststore=store=createStore(reducer);
store.dispatch(action)
store.subscribe(listener)
store.getState()
store.replaceReducer(nextReducer)
ThecreateStore()callneedsoneormorereducersgiventoit.Itcanalsobe
passed some middleware. There are other npm packages you can download to
actasmiddleware,oryoucanwriteyourown.Thisjustmeansthattheflowcan
beinterceptedandalteredinsomeway.Thisissimilartowhatmiddlewaredoes
inExpresswithNode.js.Ihavenotneededtousethiscapability.
Thedispatch()calliswhatyouusetocauseachangeinstatetohappen.The
getState() function is a way to fetch the state at that moment in time and the
subscribe()isforreceivingasynchronousnotificationsasstateischanged.
In the NewsWatcher codebase, I make the initial call to createStore()with
someprovidedreducerfunctionsandthencalldispatch()alot.Tereisalsousage
ofanothermoduleontopofReduxthatIwillsoondescribe.
Theflowofastatechange
WhatyouwouldtypicallydoindevelopingyourReactcodeistocodeupa
component and use this.state for keeping data that changes local to that
component. You might recall that you do that in a component with the
this.setState()call.Thenasyouprogressandfindthatthestateissomethingthat
needstobeshared,youmoveittotheReduxstatestorage.Don’tputeverything
inReduxasthatcanbecomeoverwhelming.
To update the Redux state, you make a call to store.dispatch(action). For
example,if acomponent wanted toupdate some stateand make thatavailable
backtoitselfandtoothercomponents,itwouldmakethatcall.Thecomponent
calling dispatch() does not need to know about who will listen for that state
change.Thisisasingledirectionofflowwiththestatechange.Therearenobi-
directionalmessagesflowingthroughReduxinanykindofconversation.
The simple usage is to preconfigure Redux with what are called reducers.
Theseactasawaytofilterdataasitflowsthrough.Dispatchcallshappenand
flow through what is called an “action” in the reducers which then causes the
particularaffectedstatetochangeandbeavailable.
Codereceives thechanges inthe state viaa store.subscribe()call.Youcan
alsogeta snapshotofthe stateata giveninstantwith thecallstore.getState().
Here is a visual representation of flow for a state change as it passes via the
Reduxstore:
Figure95-Reactstatemanagement
Here is the simplest code that can be used to demonstrate the basic flow.
Notethattheactionobjectmustatleasthaveapropertynamed“type”.Thisis
usedtofindthecorrectroutertouseforprocessing.Theotherpropertiesofthe
actionobjectareuptoyoutopassanddealwith.Thefollowingcodecreatesthe
Reduxstorewiththepassedinreducer.Thefunctionusagehasaninitialobject
passedinthatsetsthestorageofthestateproperty.
constinitialState={
count:0,
currentMsg:"HelloRedux"
}
conststartState=(state=initialState,action)=>{
switch(action.type){
case'INCREMENT':
return{
...state,
count:state.count+action.incAmount;
}
default:
returnstate;
}
}
const{createStore}=Redux;
conststore=createStore(startState);
constCounter=({
value,
onIncrement
})=>(
<div>
<h1>Howmanybugshaveyoufixedtoday?</h1>
<h1>{value}</h1>
<buttononClick={onIncrement}>+</button>
</div>
);
constrender=()=>{
ReactDOM.render(
<Counter
value={store.getState().count}
onIncrement={()=>store.dispatch({type:'INCREMENT',incAmount:1})
}
/>,
document.getElementById('root')
)
};
store.subscribe(render);
render();
Thestateforagivenreducercantakeanyformyoudefineforit.Youcansee
inthepreviouscodethatitisdefinedwithdefaultvaluestobeinitializedwith.
Reduxmakesaninitialcalltothereduceronitsownandthedefaultsareset.
Toalterthestate,adispatchcallhappensandpassesintheactionobjectwith
thetypeand valuestoconsume. IntheRedux code,the stateisimmutable,so
youcannotchangeitdirectly,youhavetomakeacloneofit(i.e.seeJavaScipt
Object.assign()usage,suchasObject.assign({},state,{users:action.users});),
or use something like the immutability-helper. This will be used in the
NewsWatcher sample application. You also need to return the complete state
objectback, as it will all be replaced. The code above simply creates the new
object from properties of the existing state using the object spread syntax of
Javascript.Thenthecountisupdated.
Forexample, if itwas an object, you replacethe whole thing, notjust one
property. With the what you saw with React component state, you have one
objectthathad independentlyupdatable propertieson it.Reduxdoes notwork
thesameway.Youcan,however,havedifferentreducers thatareindependent.
Updatingone,doesnotaffecttheother.
MostapplicationswillwanttosplitoutwhatisstoredintheReduxstorage
into separate concerns. You simply do this by calling combineReducers() to
piecethemalltogether.Eachonewillreceivealloftheactionsroutedtothem,
butitisofcourseuptothereducerstomatchwhataction.typevaluestheycare
about.YoucallcreateStorewiththiscombinedrootReducerinsteadofasingle
reducerfunction.
import{combineReducers}from'redux'
constrootReducer=combineReducers({
app,
news,
sharednews,
profile
})
UsageofReduxwithreact-redux
The Redux library is a reusable JavaScript library that can be used in
JavaScript code. It is certainly usable as it is with is four methods. There are,
however,certaincodedesignpatternsofReactthatareusedoverandover.One
ofthesepatternsinvolvesstatemanipulations.Inordertocombinethatpattern
withtheuseofRedux,inaneasytoconsumeway,therewasalibrarycreated
namedreact-redux.
Togetstarted,youneedtoinstallthenpmpackagesreduxandreact-redux.
You then need to add some code to create the single Redux store. There is a
specialReactcomponentthatisusedtowrapyourapplicationcomponentsoas
to make Redux available in all of the component hierarchy. Here is what the
index.jsfileendsuplookinglike.Thekeychangeshavebeenhighlightedinbold
font.
importReactfrom'react';
importReactDOMfrom'react-dom';
import{createStore}from'redux'
import{Provider}from'react-redux'
importreducerfrom'./reducers'
importAppfrom'./App';
import'bootstrap/dist/css/bootstrap.css';
import'bootstrap/dist/css/bootstrap-theme.css';
import'./index.css';
conststore=createStore(reducer);
ReactDOM.render(
<Providerstore={store}>
<App/>
</Provider>,document.getElementById('root')
);
The App component will be a child of the Provider component. The code
fromreact-reduxlibrarywillbeabletouseitandpassinadditionalprops.Itwill
passinapropthatisafunctionnamed“dispatch”.Thisway,youdon’tneedto
pass the Redux store to the App component. Youwon’tactually be coding up
callstostore.dispatch(),justdispatch().
Thenextthingyouneedtounderstandishowtotakeacomponentyouwrite
anddotheworktogetthestatefromthechangesthathappenintheReduxstore
andtransferthatstatetothelocalpropsonthecomponent.
The state coming from the Redux store can be used in the props on a
component. Props are then changed via the dispatch to cause the behavior to
change. You do not take Redux state and transfer it to the this.state of a
component. That would be a waste, as they both serve the same purpose and
wouldbeduplicatedandyouwouldnothaveasinglesourceoftruth.
Hereiswhatyouneedtodoinacomponenttohaveitpullinstatechanges
fromaReduxstore:
import{connect}from'react-redux'
...lotsofcomponentcodeleftout...
MyComponent.propTypes={
dispatch:PropTypes.func.isRequired
};
constmapStateToProps=state=>{
return{
session:state.app.session,
user:state.profile.user,
isLoading:state.profile.isLoading
}
}
exportdefaultconnect(mapStateToProps)(MyComponent)
Thiscodemayseemalittlestrangewhenyoufirstlookatit.Whatyouare
doing here is instead of a simple export of MyComponent, you call the react-
reduxprovidedconnect()functionandthathastheabilitytobeabletotakeyour
componentandwrapitforitsownpurposes.Reduxkeepstrackofwhenitneeds
tosendstateupdatesandre-renderitandprovidethedispatchprop.Youcansee
thatyoucangetwhateveryoulikefromtheReduxstore.Thistimeitispulling
fromtheappandprofilesettings.
Youpasstotheconnect()callacallbackfunctionthatletsyougetaccessto
theReduxstate.ThisfunctioniscalledeverytimeanythingintheReduxstate
changes. Using the state passed in, you can take whatever your particular
componentcaresabout.Thiscontainsallofthedifferentstoredstatefromallof
thereducersthatareinuse.
Thereismorethatispossible withreact-redux.Forexample,youcan pass
other parameters to connect(), such as a function that allows you to hook up
dispatch calls from your component, such as UI click handlers that make
dispatchcalls.Thensomewhereinyourcode,youcancallthepreparedfunction
thatisavailableontheprops.
import{connect}from'react-redux'
...lotsofcomponentcodeleftout...
sendAction=()=>{
this.props.sendAction()
}
render(){
<div>
<ButtononClick={sendAction}/>
</div>
}
MyComponent.propTypes={
dispatch:PropTypes.func.isRequired
};
constmapStateToProps=state=>{
return{
session:state.app.session,
user:state.profile.user,
isLoading:state.profile.isLoading
}
}
functionmapDispatchToProps(dispatch){
return({
sendAction:()=>{dispatch(
{type:'SOME_ACTION',someData:1})}
})
}
exportdefaultconnect(mapStateToProps,
mapDispatchToProps)(MyComponent)
Youdon’thavetosetthingsupthisway,asdispatch()isavailableanywayas
a prop of the component already. Here is an example that adds in the
mapDispatchToPropsfunction:
Ifyoudon’twanttosetupamapDispatchToPropsthenyoujustcalldispatch
fromthis.propsasfollows.
sendAction=()=>{
this.props.dispatch({type:'SOME_ACTION',someData:1})
}
Note: The usage of additional libraries like Redux is completely at your
discretion.Onlyusealibraryifatanypointyouseethatyoucansimplifyyour
code. Sometimes a simple code base with no design frills works just fine. You
must decide when and how to refactor and introduce new libraries into your
code and what the cost versus benefit will be. In the case of Redux, I believe
thereisaclearbenefit.YouwillseeitusedthroughouttheNewsWatchercode.
Chapter17:NewsWatcherApp
DevelopmentwithReact
With the previous chapters content, you are now able to understand the
constructionoftheNewsWatcherpresentationlayercode.Ifyoufollowedalong
inthepreviouschapterinthesectiononinstallationsteps,youareallsettobegin
writingcode.
In this chapter, you will learn about the files as they exist in the project
postedonGithubandhoweachoneispiecedtogether.Ifyouclonetheproject
fromGitHub,theprojectfolderswilllookasfollows:
Figure96-VSCodefiletree
Note: Don’t forget that you can access all of the code for the sample
NewsWatcherprojectathttps://github.com/eljamaki01/NewsWatcher2RWeb.
Youmay havenoticedthat youhad aplaceholder index.htmlfilethat your
Node.jsservice was serving up. The existing Node.js project frompart two of
this book is capable of serving up your UI after you make just a few minor
tweaks.
17.1WhereitAllStarts(src/index.js)
BackinthesectionsontheNode.jswebservicedevelopment,yousawafew
linesofcodeintheserver.jsfilethatservedupagetrequestforthemainHTML
page.ThisallowedforthedownloadinganddisplayoftheNewsWatchersite.
Theexpressroutingforthatfilewasseparatedfromthoseprovidingaccess
tothebackendAPI.Herearethelinesfromserver.jsfortheroutehandlingthat
servesuptheindex.htmlfileanditsassociatedstaticresources:
app.get('/',function(req,res){
res.sendFile(path.join(__dirname,'build','index.html'));
});
app.use(express.static(path.join(__dirname,'build')));
Theapp.get()callgivesaspecificrouteforagetrequesttoyourrootsiteand
statesthatyouwillalwaysserveupyourindex.htmlfileforthat.Thisisreally
allyouneedtosetuptheuseofReacttogetthereactapplicationsenttoberun
onaclient-sidebrowser.
Theapp.use()call sets upmiddleware for arouter path forall of thestatic
files. You simply tell it that you have this directory named “build” for where
theyallarelocated.
Youwillnotethatthisfinalindex.hmlfileisnotafileyoucreated,butitis
generatedfor youin the buildprocess fromthe index.js filethat youprovided
and the template index.html file found in the public folder. Look in the build
foldertofindthegeneratedindex.htmlfilethatgetsservedup.
Hereistheindex.jsfile.Thefewadditionsyouwouldfindherearejustthe
inclusion of css files that can be used across anything that is rendered. In
particular,youseetheonesforstylingbootstrapelements.Youalsoseethecode
tosetupReduxtobeusedacrossalltheapplication.Thereducersarebroughtin
andtheProvidercomponentissomethingfromreact-reduxthatprovidesRedux
totheoverallapplication.
importReactfrom'react';
importReactDOMfrom'react-dom';
import{createStore}from'redux'
import{Provider}from'react-redux'
importreducerfrom'./reducers'
importAppfrom'./App';
import'bootstrap/dist/css/bootstrap.css';
import'bootstrap/dist/css/bootstrap-theme.css';
import'./index.css';
conststore=createStore(reducer);
ReactDOM.render(
<Providerstore={store}>
<App/>
</Provider>,document.getElementById('root')
);
ThecalltoReactDOM.render()isthestartingpointcalltoreactthatrenders
totherootdivelement.AllofyourUIemanatesfromhere,andcomesfromthe
Appcomponent.
17.2Thehubofeverything(src/App.js)
The App.js file is where we establish the UI that renders the views. This
meansthatitprovidesthingslikethecapabilitytopresentanavigationbarand
allowsforthenavigatingbetweenthedifferentpageviews,suchasloggingin,
viewingnews,andsettingprofilesettings.ThisfilesetsuptheAppclassthatis
derivedfromtheComponentclass.
Imports,constructorandthecomponentDidMountlifecycle
Theimportsaresetuptobringinexternallibrariesandtheothercomponents
thatareneededforviewsthatareusedinthenavigation.Thenthereisthecode
fortheAppcomponentclass.
Reduxisusedforthestatesettingsthatareneeded.Youwillfindtheuseof
thedispatch()functiontostorestatedatainReduxinacentralwaythatisglobal
totheapplication.LookatthemapStateToProps()functionatthebottomandyou
willseewhatpropertiestheAppclassretrievesfromRedux.Theyare:
Thestateobjecttotellusifwearesignedin.
Thesessiontokenthatwasretrievedfromtheserver-sidelogin.
ThestatusmessagethatisdisplayedatthetopoftheUI.
The componentDidMount() lifecycle event method is where you do
processingoncetheUIhasbeenrenderedandneedtomakeanyalterations.The
codechecksthelocalbrowserstoragetoseeifatokenhasbeensavedawayto
beretrievedandifsogetsthatandsetsamessagethattheuserisloggedin.The
userwouldneedtohaveselectedtheoptionatlogintimetohaveonesaved.The
dispatchthathappensplacesthattokenintoReduxstoragetoglobalaccessfrom
allotherpagesthatneedit,suchasforprofileretrieval.
The logged in flag is set to true in that same dispatch processing, which
altersthemenurendering,asyouwillsoonsee.ThevalueofcurrentMsgisset
throughthedispatchandwouldshowupintheUI.
TheUIisrenderedoncefromtheserver,withtheinitialHTML,withitsCSS
andJavaScriptandthenitiscompletelyself-sufficientintheclientbrowserfrom
thenon.Therearenopagesbeingrenderedfromtheserversideafterthat.You
could create a combination of server and client-side page rendering in a true
isomorphicapplicationaswillbediscussedlater.Hereisthecodethathasbeen
discussedthusfar.
//App.js
importReact,{Component}from'react';
import'./App.css';
import{HashRouter,Switch,Route}from'react-router-dom'
import{Navbar,Nav,NavItem}from'react-bootstrap';
import{IndexLinkContainer}from'react-router-bootstrap';
import{connect}from'react-redux'
importLoginViewfrom'./views/loginview';
importNewsViewfrom'./views/newsview';
importHomeNewsViewfrom'./views/homenewsview';
importSharedNewsViewfrom'./views/sharednewsview';
importProfileViewfrom'./views/profileview';
importNotFoundfrom'./views/notfound';
classAppextendsComponent{
componentDidMount(){
//CheckfortokeninHTML5clientsidelocalstorage
conststoredToken=window.localStorage.getItem("userToken");
if(storedToken){
consttokenObject=JSON.parse(storedToken);
this.props.dispatch({ type: 'RECEIVE_TOKEN_SUCCESS', msg: `Signed in as
${tokenObject.displayName}`,session:tokenObject});
}else{
}
}
...codeleftout...
}
constmapStateToProps=state=>{
return{
loggedIn:state.app.loggedIn,
session:state.app.session,
currentMsg:state.app.currentMsg
}
}
exportdefaultconnect(mapStateToProps)(App)
This application is a SPA application, so hash based browser navigation is
used. This does not allow for history to be kept and if you try and use the
browserbackbutton,theapplicationwouldnothaveanyhistorytogobackand
forthfrom.
Navigationbar
Themain point of this App component is to set up the navigation bar that
will exist for users to get access to the menus and see a status message. The
DOMrenderingmakesuseofsomehandybootstrapstyling.Youcaninvestigate
theparticularsofhowitallworksonyourownthroughtheusageofthereact-
bootstrap npm package. Here is an image of what would be rendered on a
smartphonedevice.ItshowstheUIstatewhenthemenuisopened.
Figure97-Sampleimagefromasmartphone
The react-boostrap usage has the ability to do what is called “responsive”
webrenderinganditcreatesabuttonthatwillbeusedtoprovideadrop-down
menuwhen it isbeing run ona mobile smartphone.If you runit in adesktop
browser, it does not appear this way, but the navigation bar is spread out.
Bootstrapclassesadapttothesizeofthedisplay.
Note:Thegreatthingisthatyoudon’tneedtousetherawbootstrapstylings
withlow-levelHTMLelements.React-boostrapwrapsallofthatandpresentsit
foryourusageinreactComponentsthatareconsumed.
Aspartoftheheaderofthenavbar,thereisaspanelementusedtodisplay
messagesyouwanttheusertosee.Forexample,iftheirloginfails,youwantthe
usertoknowthat.ThestateissetthroughReduxwiththeappropriatemessage
string.
EachIndexLinkContainerentryrepresentsthemenuselectionsfornavigating
around the application. There is a clever way to show or hide each entry
dependingonthestateoftheapp.TheloggedInpropertyisusedtodeterminethe
displayofnavigationbarentries.Whenauserisloggedin,youwantallofthe
menuselectionsvisible.Untilthen,theyarehidden.
The actual navigation click-handling is done through the use of the react-
router-domnpmpackage.Attheverytopofwhatisrendered,istheusageofthe
HashRoutercomponent.Thisiswhatwrapseverythingandgivesustheability
torender in the DOM whatever weset up as a page view route.Thenwe can
either let the navigation bar selections control the route to select, or in some
cases,managethenavigationprogrammatically.
You can see the react-router-dom Switch component that is used in
conjunctionwiththeRoutecomponenttohandlewhatpageviewsgetrendered.
TheinterestingthingisthatforeachRoute,youspecifythepaththatitisforand
the component to render. In one case, the code has to actually use the render
Propinordertopassthestateintotheusageofthepageviewcomponent.Here
istherender()functionfortheAppcomponentthatcontrolsthenavigation:
render(){
return(
<HashRouter>
<div>
<NavbarfluiddefaultcollapseOnSelect>
<Navbar.Header>
<Navbar.Brand>
NewsWatcher{this.props.currentMsg&&<span><small>({this.props.currentMsg})</small>
</span>}
</Navbar.Brand>
<Navbar.Toggle/>
</Navbar.Header>
<Navbar.Collapse>
<Nav>
{<IndexLinkContainer to="/" replace><NavItem >Home Page News</NavItem>
</IndexLinkContainer>}
{this.props.loggedIn&&
<IndexLinkContainerto="/news"replace>
<NavItem>MyNews</NavItem>
</IndexLinkContainer>}
{this.props.loggedIn&&
<IndexLinkContainerto="/sharednews"replace>
<NavItem>SharedNews</NavItem>
</IndexLinkContainer>}
{this.props.loggedIn&&
<IndexLinkContainerto="/profile"replace>
<NavItem>Profile</NavItem>
</IndexLinkContainer>}
{this.props.loggedIn&&
<NavItemonClick={this.handleLogout}>Logout</NavItem>}
{!this.props.loggedIn&&
<IndexLinkContainerto="/login"replace>
<NavItem>Login</NavItem>
</IndexLinkContainer>}
</Nav>
</Navbar.Collapse>
</Navbar>
<hr/>
<Switch>
<Routeexactpath="/"component={HomeNewsView}/>
<Routepath="/login"component={LoginView}/>
<Routepath="/news"component={NewsView}/>
<Routepath="/sharednews"component={SharedNewsView}/>
<Route path="/profile" render={props => <ProfileView appLogoutCB={this.handleLogout}
{...props}/>}/>
<Routecomponent={NotFound}/>
</Switch>
</div>
</HashRouter>
);
}
Note: Look at the top of the top of the file App.js file to see the import
statements. These will help you keep straight which components come from
which npm packages. For example, those from react-bootstrapand those from
react-router-dom.
Loggingout
Thenavigationbarhasaselectiontologtheuserout.Thissimplyplacesa
calltothebackendserviceanditcandowhateveritwouldlikecodewise,but
theclientsidereallyjustneedstosetthestatetoreflectthatandredirecttheview
to the login. This is done through a browser capability with the use of setting
window.location.hash.
handleLogout=(event)=>{
const{dispatch}=this.props
event&&event.preventDefault();
fetch(`/api/sessions/${this.props.session.userId}`,{
method:'DELETE',
headers:newHeaders({
'x-auth':this.props.session.token
}),
cache:'default'//no-storeorno-cache?
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
dispatch({type:'DELETE_TOKEN_SUCCESS',msg:"Signedout"});
window.localStorage.removeItem("userToken");
window.location.hash="";
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',msg:`Signoutfailed:${error.message}`});
});
}
17.3ReduxReducers(src/reducers/index.js)
You saw in the src/index.js file how the reducers were brought into the
application when it started up. The line was as follows that set up the Redux
Store:
importreducerfrom'./reducers'
conststore=createStore(reducer);
Incaseyouforgot,ifyouhavearequireorimportstatementofadirectory,it
willbydefaultlookforafilenamedindex.jsandusethat.
Index.jsthencombinesalltheseparatereducersintooneandexportsthat:
//src/reducers/index.js
import{combineReducers}from'redux'
importappfrom'./app'
importhomenewsfrom'./homenews'
importnewsfrom'./news'
importsharednewsfrom'./sharednews'
importprofilefrom'./profile'
constrootReducer=combineReducers({
app,
homenews,
news,
sharednews,
profile
})
exportdefaultrootReducer
Let’snowlookatafewofthereducersbringcombined.
TheAppReducer
The App reducer has three actions that it supports MSG_DISPLAY,
RECEIVE_TOKEN_SUCCESS and DELETE_TOKEN_SUCCESS. We have
seen these used in the src/app.js file where the redux state is used for each of
these.InthecaseoftheMSG_DISPLAY,itsimplysetsanewstringforthetext
ofthecurrentMsgproperty.
Notice the “…state” code that is a JavaScript way of taking an object and
pullinginallofitsproperties.Thisway,youdon’thavetolistthem.Thisway
you get whatever was set for session and loggedIn and then currentMsg is
overridden.
//src/reducers/app.js
constinitialState={
loggedIn:false,
session:null,
currentMsg:""
}
constappLevel=(state=initialState,action)=>{
switch(action.type){
case'MSG_DISPLAY':
return{
...state,
currentMsg:action.msg
}
case'RECEIVE_TOKEN_SUCCESS':
return{
...state,
loggedIn:true,
session:action.session,
currentMsg:action.msg
}
case'DELETE_TOKEN_SUCCESS':
return{
...state,
loggedIn:false,
session:null,
currentMsg:action.msg
}
default:
returnstate
}
}
exportdefaultappLevel
Iwillshowonemorereducer.Thisonewillillustratethepointthatthestate
in the Redux store is immutable, so you have to either replace the complete
object,oruseaspecialmoduletohelpyoumanagethat.
Thereisannpmmodulenamedimmutability-helperthatallowsyoutomake
achangetoanimmutableobject.Partofthisstatesettingisanewspropertythat
isanarrayofnewsstorieswiththeircomments.
AnewcommentisaddedwiththeADD_COMMENT_SUCCESSaction.It
doesthatbypushingittotheendofthearrayforanewsstory.Inotherwords,
thereisanarrayofnewsstoriesandeachofthosehaveanarrayofcomments.If
we did not use the immutability helper, a completely new copy of the array
wouldhavetobemadeandreplacedeachtime.
//src/reducers/sharednews.js
importupdatefrom'immutability-helper';
constinitialState={
isLoading:true,
news:null
}
constnews=(state=initialState,action)=>{
switch(action.type){
case'REQUEST_SHAREDNEWS':
return{
isLoading:true,
news:[]
}
case'RECEIVE_SHAREDNEWS_SUCCESS':
return{
//...state,
isLoading:false,
news:action.news,
}
case'ADD_COMMENT_SUCCESS':
return{
...state,
news:update(state.news,{
[action.storyIdx]:{
comments:{
$push:[{displayName:action.displayName,comment:action.comment}]
}
}
})
}
default:
returnstate
}
}
exportdefaultnews
Theimmutabilityhelpersyntaxisabittrickyhere.Whatisbeingalteredis
an array property found in an object in the new array. The first thing to do is
selecttheindexofthestoryinthenewarray,andthentopushacommenttothe
commentsarrayfoundinthatobjectentry.
Alessefficientwaywouldbetoclonethearrayandthenalterit.Somearray
functionslikeslice()actually returnanewarrayand don’tmutatetheold one.
Youcanthusgetthenewarrayandthenalteritasyoulike.
Youcan use the JavaScipt Object.assign() that allows you to create a new
objectfromtheexistingstateobjectandthenoverridethecommentspropertyof
that.Hereishowthecodewouldbeforanarrayusingtheslice()function:
case'ADD_COMMENT_SUCCESS':
varnewNews=state.news.slice(0);
newNews[action.storyIdx].comments.push({ displayName: action.displayName, comment:
action.comment});
return{
...state,
news:newNews
}
Therestofthesectionsinthischaptergooverthecodethatexistsineachof
theviewsthatarerenderedlogin,homeNews,news,sharednewsandprofile.
Inthe previous chapterson React fundamentals,you learned about Reduxand
refactoringyourcomponenttosplitthemintocontrollerandviewcomponents.I
have chosen not to complicate the code by splitting out each view into its
controller/containercomponent.Ikeepallthecodeinonesinglecomponent.
17.4TheLoginPage(src/views/loginview.js)
To log a user in, their email and password are entered and verified by the
backendservice.Theloginpagecontainsacheckboxfortheusertospecifythat
theywanttheirlocaldevicetostoretheirlogintokenforthem.Iftheuserisnot
registeredyet,theycanclicktobringupapopupmodaldialogformtoregistera
newaccount.Aspartoftheregistration,anemailneedstobeprovided.Emails
areuniqueinthesystem,sonoduplicatesareallowedinthedatalayer.Hereis
whattheUIlookslikeforloggingin.
Figure98-NewsWatcherloginform
The HTML code for the form is set up to have a submit handler. The
handleLogin()functionofthe componentis thecode thatexecutesat thattime.
To get the data from the form, various Bootstrap components are used. These
bindtheirdatatothestateofthecomponentthroughonChangehandlersthatget
setaseachcharacteristyped.Hereistherendercodefortheloginpageasfound
intheloginview.jsfile:
render(){
//Ifalreadyloggedin,don'tgohereandgetroutedtothenewsview
if(this.props.session){
returnnull;
}
return(
<div>
<formonSubmit={this.handleLogin}>
<FieldGroup
id="formControlsEmail2"
type="email"
glyph="user"
label="EmailAddress"
placeholder="Enteremail"
onChange={this.handleEmailChange}
/>
<FieldGroup
id="formControlsPassword2"
glyph="eye-open"
label="Password"
type="password"
onChange={this.handlePasswordChange}
/>
<Checkboxchecked={this.state.remeberMe}
onChange={this.handleCheckboxChange}>
Keepmeloggedin
</Checkbox>
<ButtonbsStyle="success"bsSize="lg"blocktype="submit">
Login
</Button>
</form>
<p>NotaNewsWatcheruser?
<astyle={{cursor:'pointer'}}
onClick={this.handleOpenRegModal}>SignUp</a>
</p>
{this._renderRegisterModal()}
</div>
);
}
You can see that at the top of the render() function we first see if we are
loggedin.Whenweareloggedin,thesessionpropertyfromReduxisset.
Theotherveryinterestingthingthatisgoingonisthatyouseethecalltothe
function renderRegisterModal(). I could have just placed all the HTML right
there in the render function. I did not do that as that function is large enough
already.Breakingit out makesthe codeeasier toread. Thistakes that outand
makesitself-contained.
TheotherthingtoknowisthatthisUIforregisteringauserofNewsWatcher
is a modal dialog, and uses the <Modal> component that is provided by
BootstrapforReact.Itisalwaysrendered,butitisbeingshownorhiddenbya
statepropertyandyoucanseethatitstartsoutasbeinghidden.
ThemodalregistrationdialogUIlooksasfollows:
Figure99-NewsWatcherRegistrationModal
HereisthecodefortheregistrationmodalUI:
_renderRegisterModal=()=>{
return(<Modalshow={this.state.showModal}onHide={this.handleCloseRegModal}>
<Modal.HeadercloseButton>
<Modal.Title>Register</Modal.Title>
</Modal.Header>
<Modal.Body>
<formonSubmit={this.handleRegister}>
<FieldGroup
id="formControlsName"
type="text"
glyph="user"
label="DisplayName"
placeholder="Enterdisplayname"
onChange={this.handleNameChange}
/>
<FieldGroup
id="formControlsEmail"
type="email"
glyph="user"
label="EmailAddress"
placeholder="Enteremail"
onChange={this.handleEmailChange}
/>
<FieldGroup
id="formControlsPassword"
glyph="eye-open"
label="Password"
type="password"
onChange={this.handlePasswordChange}
/>
<ButtonbsStyle="success"bsSize="lg"blocktype="submit">
<Glyphiconglyph="off"/>Register
</Button>
</form>
</Modal.Body>
<Modal.Footer>
<Button bsStyle="danger" bsSize="default" onClick={this.handleCloseRegModal}><Glyphicon
glyph="remove"/>Cancel</Button>
</Modal.Footer>
</Modal>)
}
ThisUIuses aformand hasa functionthathandles thesubmit.Let’snow
lookatthecodethatsupportstheLoginViewcomponent.
ComponentsupportingcodeofLoginView
Thereistheusualconstructorthatsetsupwhatisneededforthestate.Inthis
case, we have properties that will hold the bound data from the form, such as
email and password. The showModal property is used to show and hide the
modalregistrationform.
As explained, at the top of the render function is a test to see if there is a
session token and a null is returned, which React sees and ignores. The login
menu item is not shown anyway if the user is already logged in. The session
tokencomesthroughtheusageofRedux.
Oncetheusertypesintheirusernameandpasswordandthenclick“Login”,
thehandleLogin()methodmakesthecalltothebackendtogetatoken.Itsetsthe
email and password in its HTTP POST request. If successful, the message
RECEIVE_TOKEN_SUCCESS is sent through Redux with a dispatch() call.
Thissetsthesessiontokenfortherestoftheapplicationtosee.Then,thereisa
changemadetothebrowserlocationtogotothenewspageview.
ThehandleRegistration()method makes the callto the backend to create a
useraccountinthedatalayer.Theothermethodsyouseeareforthebindingof
thedatatothestateandtheopeningandclosingofthemodalregistrationdialog
viathestatepropertyforthat.
Here is the rest of the code, minus the render code that you have already
seen:
importReact,{Component}from'react';
importReact,{Component}from'react';
importPropTypesfrom'prop-types';
import{Checkbox,Button,Modal,Glyphicon}from'react-bootstrap';
import{connect}from'react-redux'
importsuperagentfrom'superagent';
importnoCachefrom'superagent-no-cache';
import{FieldGroup}from'../utils/utils';
import'../App.css';
classLoginViewextendsComponent{
constructor(props){
super(props);
this.state={
name:"",
email:"",
password:"",
remeberMe:false,
showModal:false
};
}
handleRegister=(event)=>{
const{dispatch}=this.props
event.preventDefault();
returnfetch('/api/users',{
method:'POST',
headers:newHeaders({
'Content-Type':'application/json'
}),
cache:'default',//no-storeorno-cacherodefault?
body:JSON.stringify({
displayName:this.state.name,
email:this.state.email,
password:this.state.password
})
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==201){
thrownewError(response.json.message);
}
dispatch({type:'MSG_DISPLAY',msg:"Registered"});
this.setState({showModal:false});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',msg:`Registrationfailure:${error.message}`});
});
}
handleLogin=(event)=>{
const{dispatch}=this.props
event.preventDefault();
returnfetch('/api/sessions',{
method:'POST',
headers:newHeaders({
'Content-Type':'application/json'
}),
cache:'default',//no-storeorno-cacherodefault?
body:JSON.stringify({
email:this.state.email,
password:this.state.password
})
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==201){
thrownewError(response.json.message);
}
//Setthetokeninclientsidestorageiftheuserdesires
if(this.state.remeberMe){
varxfer={
token:response.json.token,
displayName:response.json.displayName,
userId:response.json.userId
};
window.localStorage.setItem("userToken",JSON.stringify(xfer));
}else{
window.localStorage.removeItem("userToken");
}
dispatch({ type: 'RECEIVE_TOKEN_SUCCESS', msg: `Signed in as
${response.json.displayName}`,session:response.json});
window.location.hash="#news";
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',msg:`Signinfailed:${error.message}`});
});
}
handleNameChange=(event)=>{
this.setState({name:event.target.value});
}
handleEmailChange=(event)=>{
this.setState({email:event.target.value});
}
handlePasswordChange=(event)=>{
this.setState({password:event.target.value});
}
handleCheckboxChange=(event)=>{
this.setState({remeberMe:event.target.checked});
}
handleOpenRegModal=(event)=>{
this.setState({showModal:true});
}
handleCloseRegModal=(event)=>{
this.setState({showModal:false});
}
...render()and_renderRegisterModal()leftout...
LoginView.propTypes={
dispatch:PropTypes.func.isRequired,
session:PropTypes.object
};
constmapStateToProps=state=>{
return{session:state.app.session}
}
exportdefaultconnect(mapStateToProps)(LoginView)
17.5DisplayingtheNews(src/views/newsview.jsand
src/views/homenewsview.js)
ThisNewsViewcomponentdisplaysthefilterednewspage.TheUIlooksas
follows:
Figure100-NewsWatchernewspage
There is a dropdown list that displays the list of news filters to select
between.TheJavaScriptmap()functionisusedtogothroughthearrayoffilters
and populate that dropdown. Each one is given text from the filter.name
property.ThereisanotheruseofthenewsFilterarraywiththemap()functionto
rendereachofthenewsstoriesfortheselectedfilter.Alinkisprovidedforeach
story,alongwiththeimageandURLtoclickandopen.Thereisalotofusageof
Bootstrapcomponentsinthiscode.Hereistherendermethod:
render(){
if(this.props.isLoading){
return(
<h1>Loadingnews...</h1>
);
}
return(
<div>
<h1>News</h1>
<FormGroupcontrolId="formControlsSelect">
<FormControlbsSize="lg"componentClass="select"
placeholder="select"
onChange={this.handleChangeFilter}
value={this.state.selectedIdx}>
{this.props.newsFilters.map((filter,idx)=>
<optionkey={idx}value={idx}>
<strong>{filter.name}</strong>
</option>
)}
</FormControl>
</FormGroup>
<hr/>
<Media.List>
{this.props.newsFilters[this.state.selectedIdx].newsStories.map((story,idx)=>
<Media.ListItemkey={idx}>
<Media.Left>
<ahref={story.link}target="_blank">
<imgalt=""className="media-object"src={story.imageUrl}/>
</a>
</Media.Left>
<Media.Body>
<Media.Heading><b>{story.title}</b></Media.Heading>
<p>{story.contentSnippet}</p>
{story.source}<span>{story.hours}</span>
<Media.Body>
<a style={{ cursor: 'pointer' }} onClick={(event) => this.handleShareStory(idx,
event)}>Share</a>
</Media.Body>
</Media.Body>
</Media.ListItem>
)}
<Media.ListItemkey={999}>
<Media.Left>
<ahref="http://developer.nytimes.com"target="_blank"
rel="noopenernoreferrer">
<imgalt=""src="poweredby_nytimes_30b.png"/>
</a>
</Media.Left>
<Media.Body>
<Media.Heading><b>DataprovidedbyTheNewYorkTimes</b></Media.Heading>
</Media.Body>
</Media.ListItem>
</Media.List>
</div>
);
}
ThereisalsotheabilitytoclickonalinktoSharestories.
Componentsupportingcode
Whenthecomponentisopened,codeisruntodothefetchingofthenews
stories.ThisisdoneinthecomponentDidMount()method.Youcanalsoseeat
thetop ofthe codehow theconstructor setsup thestate forthe selectednews
filter.TheisLoadingpropertycomesfromReduxandisusedforaUIindication
thatthenewsstoriesarebeingfetched.Themessagegoesawayoncethedatais
available.ThehandleShareStory()methodisprovidedsothatastorycanbesent
tothebackendtobeputinacommonlocationforalluserstoseeandcomment
on.
ThetoHours() is this little helper function that is being used to format the
text to display how old a news story is. Here is the code for the newsview.js,
withtherendermethodleftout,asthatwasalreadyshown:
importReact,{Component}from'react';
importPropTypesfrom'prop-types';
import{FormGroup,FormControl,Media}from'react-bootstrap';
import{connect}from'react-redux'
importsuperagentfrom'superagent';
importnoCachefrom'superagent-no-cache';
import{toHours}from'../utils/utils';
import'../App.css';
classNewsViewextendsComponent{
constructor(props){
super(props);
this.state={
selectedIdx:0
};
}
componentDidMount(){
if(!this.props.session){
returnwindow.location.hash="";
}
const{dispatch}=this.props
dispatch({type:'REQUEST_NEWS'});
fetch(`/api/users/${this.props.session.userId}`,{
method:'GET',
headers:newHeaders({
'x-auth':this.props.session.token
}),
cache:'default'//no-storeorno-cache?
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
for(vari=0;i<response.json.newsFilters.length;i++){
for(varj=0;j<
response.json.newsFilters[i].newsStories.length;j++)
{
response.json.newsFilters[i].newsStories[j].hours =
toHours(response.json.newsFilters[i].newsStories[j].date);
}
}
dispatch({type:'RECEIVE_NEWS_SUCCESS',newsFilters:response.json.newsFilters});
dispatch({type:'MSG_DISPLAY',msg:"Newsfetched"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',msg:`Newsfetchfailed:${error.message}`});
});
}
handleChangeFilter=(event)=>{
this.setState({selectedIdx:parseInt(event.target.value,10)});
}
handleShareStory=(index,event)=>{
const{dispatch}=this.props
event.preventDefault();
fetch('/api/sharednews',{
method:'POST',
headers:newHeaders({
'x-auth':this.props.session.token,
'Content-Type':'application/json'
}),
cache:'default',//no-storeorno-cacherodefault?
body:JSON.stringify(
this.props.newsFilters[this.state.selectedIdx].newsStories[index])
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==201){
thrownewError(response.json.message);
}
dispatch({type:'MSG_DISPLAY',msg:"Storyshared"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',
msg:`Shareofstoryfailed:${error.message}`});
});
}
render(){
...TAKENOUT...ALREADYSHOWN...
}
}
NewsView.propTypes={
dispatch:PropTypes.func.isRequired
};
constmapStateToProps=state=>{
return{
session:state.app.session,
newsFilters:state.news.newsFilters,
isLoading:state.news.isLoading
}
}
exportdefaultconnect(mapStateToProps)(NewsView)
Thehomenewsstorypageisjustastrippeddownversionofthecodeseen
above, so there is no need to explain it. That is found in the HomeNewsView
component.
17.6SharedNewsPage(src/views/sharednewsview.js)
Thesharednewsstoryviewhasthesametypeofnewslistingcapabilityyou
haveseenbefore.Hereistherendercode:
render(){
if(this.props.isLoading){
return(<h1>Loadingsharednews...</h1>);
}
return(
<div>
<h1>SharedNews</h1>
<Media.List>
{this.props.news.map((sharedStory,idx)=>
<Media.ListItemkey={idx}>
<Media.Left>
<ahref={sharedStory.story.link}target="_blank">
<imgalt=""className="media-object"
src={sharedStory.story.imageUrl}/>
</a>
</Media.Left>
<Media.Body>
<Media.Heading><b>{sharedStory.story.title}</b></Media.Heading>
<p>{sharedStory.story.contentSnippet}</p>
{sharedStory.story.source}–
<span>{sharedStory.story.hours}</span>
<astyle={{cursor:'pointer'}}onClick={(event)=>
this.handleOpenModal(idx,event)}>Comments</a>
</Media.Body>
</Media.ListItem>
)}
<Media.ListItemkey={999}>
<Media.Left>
<ahref=http://developer.nytimes.com
target="_blank"rel="noopenernoreferrer">
<imgalt=""src="poweredby_nytimes_30b.png"/>
</a>
</Media.Left>
<Media.Body>
<Media.Heading><b>DataprovidedbyTheNewYorkTimes</b></Media.Heading>
</Media.Body>
</Media.ListItem>
</Media.List>
{this.props.news.length>0&&
<Modalshow={this.state.showModal}onHide={this.handleCloseModal}>
<Modal.HeadercloseButton>
<Modal.Title>AddComment</Modal.Title>
</Modal.Header>
<Modal.Body>
<formonSubmit={this.handleAddComment}>
<FormGroupcontrolId="commentList">
<ControlLabel><Glyphiconglyph="user"/>
Comments</ControlLabel>
<ulstyle={{height:'10em',overflow:'auto',
'overflow-x':'hidden'}}>
{this.props.news[this.state.selectedStoryIdx].
comments.map(comment=>
<li>
<div>
<p>'{comment.comment}'-{comment.displayName}
</p>
</div>
</li>
)}
</ul>
</FormGroup>
{this.props.news[this.state.selectedStoryIdx].
comments.length<30&&
<div>
<FieldGroup
id="formControlsComment"
type="text"
glyph="user"
label="Comment"
placeholder="Enteryourcomment"
onChange={this.handleCommentChange}
/>
<Buttondisabled={this.state.comment.length===0}
bsStyle="success"bsSize="lg"
blocktype="submit">
<Glyphiconglyph="off"/>Add
</Button>
</div>
}
</form>
</Modal.Body>
<Modal.Footer>
<ButtonbsStyle="danger"bsSize="default"
onClick={this.handleCloseModal}>
<Glyphiconglyph="remove"/>Close
</Button>
</Modal.Footer>
</Modal>
}
</div>
);
}
Thereisacapabilitytocommentoneachstory.Thatisdonethroughamodal
dialog similar to the one used for user registration. Here is the image of the
viewingofcomments:
Figure101-NewsWatcheraddcommentUI
Componentsupportingcode
ThiscodeisverysimilartotheNewsViewcomponent.Thedifferenceisin
thecodeforaddinganewcomment.Hereisthecode:
importReact,{Component}from'react';
importPropTypesfrom'prop-types';
import{FormGroup,ControlLabel,Button,Modal,Glyphicon,Media}from'react-bootstrap';
import{connect}from'react-redux'
importsuperagentfrom'superagent';
importnoCachefrom'superagent-no-cache';
import{FieldGroup,toHours}from'../utils/utils';
import'../App.css';
classSharedNewsViewextendsComponent{
constructor(props){
super(props);
this.state={
comment:"",
selectedStoryIdx:0
};
}
componentDidMount(){
if(!this.props.session){
returnwindow.location.hash="";
}
const{dispatch}=this.props
dispatch({type:'REQUEST_SHAREDNEWS'});
fetch('/api/sharednews',{
method:'GET',
headers:newHeaders({
'x-auth':this.props.session.token
}),
cache:'default'//no-storeorno-cache?
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
for(vari=0;i<response.json.length;i++){
response.json[i].story.hours=
toHours(response.json[i].story.date);
}
dispatch({type:'RECEIVE_SHAREDNEWS_SUCCESS',news:response.json});
dispatch({type:'MSG_DISPLAY',msg:"SharedNewsfetched"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',
msg:`SharedNewsfetchfailed:${error.message}`});
});
}
handleOpenModal=(index,event)=>{
this.setState({selectedStoryIdx:index,showModal:true});
}
handleCloseModal=(event)=>{
this.setState({showModal:false});
}
handleAddComment=(event)=>{
const{dispatch}=this.props
event.preventDefault();
fetch(`/api/sharednews/${this.props.news[this.state.selectedStoryIdx].story.storyID}/Comments`,{
method:'POST',
headers:newHeaders({
'x-auth':this.props.session.token,
'Content-Type':'application/json'
}),
cache:'default',//no-storeorno-cacherodefault?
body:JSON.stringify({comment:this.state.comment})
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==201){
thrownewError(response.json.message);
}
varstoryIdx=this.state.selectedStoryIdx;
dispatch({type:'ADD_COMMENT_SUCCESS',comment:this.state.comment,
displayName:this.props.session.displayName,
storyIdx:storyIdx});
this.setState({showModal:false,comment:""});
dispatch({type:'MSG_DISPLAY',msg:"Commentadded"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',
msg:`Commentaddfailed:${error.message}`});
});
}
handleCommentChange=(event)=>{
this.setState({comment:event.target.value});
}
render(){
...TAKENOUT...ALREADYSHOWN...
}
}
SharedNewsView.propTypes={
dispatch:PropTypes.func.isRequired
};
constmapStateToProps=state=>{
return{
session:state.app.session,
news:state.sharednews.news,
isLoading:state.sharednews.isLoading
}
}
exportdefaultconnect(mapStateToProps)(SharedNewsView)
17.7ProfilePage(src/views/profileview.js)
The profile page allows the user to create one or more news filters. Each
filterhasatitleandalistofkeywords.Youhavethesametypeofbuttondrop-
down you have seen before. Then you have the form and the three buttons to
save,delete,andcreateanewfilter.TheimageandHTMLareasfollows:
Figure102-NewsWatcherNewsFilterdialog
Thereisalinktoallowtheusertodeletetheiraccount.TheHTMLprovides
amodaldialoglikeyouhaveusedbefore.Hereistheimage:
Figure103-NewsWatcherunregisterdialog
render(){
if(this.props.isLoading){
return(
<h1>Loadingprofile...</h1>
);
}
return(
<div>
<h1>Profile:NewsFilters</h1>
<FormGroupcontrolId="formControlsSelect">
<FormControlbsSize="lg"componentClass="select"
placeholder="select"
onChange={this.handleChangeFilter}
value={this.state.selectedIdx}>
{this.props.user.newsFilters.map((filter,idx)=>
<optionkey={idx}
value={idx}><strong>{filter.name}</strong>
</option>
)}
</FormControl>
</FormGroup>
<hr/>
<form>
<FieldGroup
id="formControlsName"
type="text"
label="Name"
placeholder="NewFilter"
onChange={this.handleNameChange}
value={this.props.user.
newsFilters[this.state.selectedIdx].name}
/>
<FieldGroup
id="formControlsKeywords"
type="text"
label="Keywords"
placeholder="Keywords"
onChange={this.handleKeywordsChange}
value={this.props.user.
newsFilters[this.state.selectedIdx].keywordsStr}
/>
<divclass="btn-groupbtn-group-justified"role="group"
aria-label="...">
<ButtonToolbar>
<ButtonbsStyle="primary"bsSize="default"
onClick={this.handleAdd}><Glyphiconglyph="plus"/>Add
</Button>
<ButtonbsStyle="primary"bsSize="default"
onClick={this.handleDelete}><Glyphiconglyph="trash"/>Delete
</Button>
<ButtonbsStyle="primary"bsSize="default"
onClick={this.handleSave}><Glyphiconglyph="save"/>Save
</Button>
</ButtonToolbar>
</div>
</form>
<hr/>
<p>NolongerhaveaneedforNewsWatcher?<aid="deleteLink"
style={{cursor:'pointer'}}
onClick={this.handleOpenModal}>DeleteyourNewsWatcherAccount</a>
</p>
<Modalshow={this.state.showModal}onHide={this.handleCloseModal}>
<Modal.HeadercloseButton>
<Modal.Title>Un-Register</Modal.Title>
</Modal.Header>
<Modal.Body>
<formonSubmit={this.handleUnRegister}>
<Checkboxchecked={this.state.deleteOK}
onChange={this.handleCheckboxChange}>
CheckifyouaresureyouwanttodeleteyourNewsWatcheraccount
</Checkbox>
<Buttondisabled={!this.state.deleteOK}bsStyle="success"
bsSize="lg"blocktype="submit">
<Glyphiconglyph="off"/>DeleteNewsWatcherAccount
</Button>
</form>
</Modal.Body>
<Modal.Footer>
<ButtonbsStyle="danger"bsSize="default"
onClick={this.handleCloseModal}><Glyphiconglyph="remove"/>
Cancel
</Button>
</Modal.Footer>
</Modal>
</div>
);
}
Componentsupportingcode
All the functions necessary to provide the functionality behind the button
clicks are made available and should look very similar to code you have seen
before.
importReact,{Component}from'react';
importPropTypesfrom'prop-types';
import { FormGroup, FormControl, Checkbox, Button, Modal, Glyphicon, ButtonToolbar } from
'react-bootstrap';
import{connect}from'react-redux'
importsuperagentfrom'superagent';
importnoCachefrom'superagent-no-cache';
import{FieldGroup}from'../utils/utils';
import'../App.css';
classProfileViewextendsComponent{
constructor(props){
super(props);
this.state={
deleteOK:false,
selectedIdx:0,
};
}
componentDidMount(){
if(!this.props.session){
returnwindow.location.hash="";
}
const{dispatch}=this.props
dispatch({type:'REQUEST_PROFILE'});
fetch(`/api/users/${this.props.session.userId}`,{
method:'GET',
headers:newHeaders({
'x-auth':this.props.session.token
}),
cache:'default'//no-storeorno-cache?
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
for(vari=0;i<response.json.newsFilters.length;i++){
response.json.newsFilters[i].keywordsStr=
response.json.newsFilters[i].keyWords.join(',');
}
dispatch({type:'RECEIVE_PROFILE_SUCCESS',user:response.json});
dispatch({type:'MSG_DISPLAY',msg:"Profilefetched"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',
msg:`Profilefetchfailed:${error.message}`});
});
}
handleUnRegister=(event)=>{
const{dispatch}=this.props
event.preventDefault();
fetch(`/api/users/${this.props.session.userId}`,{
method:'DELETE',
headers:newHeaders({
'x-auth':this.props.session.token
}),
cache:'default'//no-storeorno-cache?
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
this.props.appLogoutCB();
dispatch({type:'MSG_DISPLAY',msg:"Accountdeleted"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',
msg:`Accountdeletefailed:${error.message}`});
});
}
handleNameChange=(event)=>{
this.props.dispatch({type:'ALTER_FILTER_NAME',
filterIdx:this.state.selectedIdx,value:event.target.value});
}
handleKeywordsChange=(event)=>{
this.props.dispatch({type:'ALTER_FILTER_KEYWORDS',
filterIdx:this.state.selectedIdx,value:event.target.value});
}
handleOpenModal=(event)=>{
this.setState({showModal:true});
}
handleCloseModal=(event)=>{
this.setState({showModal:false});
}
handleChangeFilter=(event)=>{
this.setState({selectedIdx:parseInt(event.target.value,10)});
}
handleAdd=(event)=>{
const{dispatch}=this.props
event.preventDefault();
if(this.props.user.newsFilters.length===5){
dispatch({type:'MSG_DISPLAY',
msg:"NomorenewsFiltersallowed"});
}else{
varlen=this.props.user.newsFilters.length;
dispatch({type:'ADD_FILTER'});
this.setState({selectedIdx:len});
}
}
handleDelete=(event)=>{
event.preventDefault();
this.props.dispatch({type:'DELETE_FILTER',
selectedIdx:this.state.selectedIdx});
this.setState({selectedIdx:0});
}
handleSave=(event)=>{
const{dispatch}=this.props
event.preventDefault();
fetch(`/api/users/${this.props.session.userId}`,{
method:'PUT',
headers:newHeaders({
'x-auth':this.props.session.token,
'Content-Type':'application/json'
}),
cache:'default',//no-storeorno-cacherodefault?
body:JSON.stringify(this.props.user)
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
dispatch({type:'MSG_DISPLAY',msg:"Profilesaved"});
})
.catch(error=>{
dispatch({type:'MSG_DISPLAY',
msg:`Profilesavefailed:${error.message}`});
});
}
handleCheckboxChange=(event)=>{
this.setState({deleteOK:event.target.checked});
}
render(){
...TAKENOUT...ALREADYSHOWN...
}
}
ProfileView.propTypes={
appLogoutCB:PropTypes.func.isRequired,
dispatch:PropTypes.func.isRequired
};
constmapStateToProps=state=>{
return{
session:state.app.session,
user:state.profile.user,
isLoading:state.profile.isLoading
}
}
exportdefaultconnect(mapStateToProps)(ProfileView)
You can see that there is code there to limit the number of filters to five.
Whatdoyousupposewouldhappeniftheuserusedthebrowserdevelopertools
tomesswiththatcodeandthenaddedthousandsoffilters?Itcouldcreateavery
largedocumentsinMongoDB.BeawarethatyoucannoteverrelyonyourUI
side bounds-checking code to do the right thing, as it is subject to tampering.
Thus,youhavetoplacecodein yourservicelayerthatwillonlytakethe first
fivefilters.
Itisalsonoteworthy,howthefunctiontohandletheloggingoutisnotinthis
code.Instead,theAppComponentthatusestheProfileViewcomponentpasses
inafunctionasapropertyonthepropsandthenthecallbackisusedtogetat
codeinApp.jsforthat.Thisseemedtobemoreappropriateforthehighercalling
codetocontrolthatandnotneedtoduplicatecodeinthiscase.
17.8NotFoundPage(src/views/notfound.js)
TheNotFoundpageissomethingthatwouldbedisplayediftheusertriedto
navigateby changingthe URLin the browserto somepage thatdid not exist.
You can see this in action if you try and go to
https://www.newswatcher2rweb.com/#/blah.HereisthatComponentcode.
importReact,{Component}from'react';
import'../App.css';
classNotFoundextendsComponent{
render(){
return(
<div>
<h3>404pagenotfound</h3>
<p>Thepageyouarelookingfordoesnotexist.</p>
</div>
);
}
}
exportdefaultNotFound;
This completes the discussion of each of the files and their corresponding
controllercode.EverythingcanbezippedupanddeployedtotheAWSElastic
Beanstalkenvironmentandused.OnWindows,youwouldselectthefoldersand
files as shown before and then right click and select Send to->Compressed
(zipped)folder.
Chapter18:UITestingofNewsWatcher
With the NewsWatcher application code completed and running, you will
wanttothoroughlytestitwitheveryconceivablescenario.Manuallydoingthis
isgreatforyourentertainment,butitwillsoongrowtediousifyouhavethatas
your only way of finding bugs. For example, what happens if you make any
changestoyourcodebase?YouhavethefunctionandloadtestsoftheAPI,but
you still need to run the UI through its paces to exercise the React JavaScript
code. Not to fear, there are many great solutions to this problem and I will
presentthisaswellasgivegeneraltipsfordebuggingthecodeinChrome.
Therearemanydifferenttechnologiesbeingusedinthecodeandeachhas
techniques for testing it in detail. For example, the React Router usage and
Redux code can be independently tested. Each individual component can be
testedinwaysthatinstantiatethemandverifycorrectDOMelementsarepresent
with the correct attributes and properties. Then you can also try testing
frameworksthatdoanautomatedrun throughofthecompleteapplicationina
browser.YouuseIDsofeachUIelementtoidentifywhattoclickorinspect.
18.1UITestingwithSelenium
There are tools you can use to record your interactions within a browser
session and then play those back. Some tools record screen locations of your
clicksandthenrelyontheUIelementstobeinthatexactsamelocationlaterfor
theinteractiontowork.OthertoolsunderstandtheDOMandcanfindelements
youspecifytoclickon.
One tool you can use to do automation testing of a UI, is a tool created a
while a while ago named Selenium. It was built so long ago that it did not
support JavaScript directly. Luckily, someone did the work to write a node.js
modulethatexposesitscapabilities.
Thefirststeptowardsaccomplishingyour UIautomationtestingwillbeto
installthe“selenium-webdriver”fromNPMtohaveitlocalinyourproject.Be
awarethatthismightnotbeassimpleasitsounds.Payattentiontotheoutput
windowasyouinstallit.Itmightfailbecauseofdependencies.Ihadtoinstall
the JDK, Python tools, and even a more complete Visual Studio install with
someoftheC++tools.Iwouldnothavedreamedthosewouldberequired.Your
experiencemightbedifferentonadifferentOS.
You might also check out an NPM module named Protractor. This is
somethingwrittenontopofselenium-webdriver.Idecidedtogodirectlythrough
theselenium-webdrivermodule,sodidnotuseProtractor.
Theselenium-webdriverisjustanSDKtoaccessthebrowserDOMandthus
you still need some type of testing framework to organize and run your tests.
Mochaisperfectfordoingthis.Thisiswhatyouwouldrunfromthecommand
prompttogetyourUIautomationtestsrunning.
.\node_modules\.bin\mocha--timeout30000ui_automation_UAT.js
You will populate a Mocha test file with code that uses the selenium-
webdrivercapabilities.Youalsoneedtosetuptherequirestatementsforasserts
andformakingHTTPrequesttodosomeneededcleanupafteratestrun.Here
aretherequirestatements:
varassert=require('assert');
varwebdriver=require('selenium-webdriver');
varrequest=require('supertest')('http://localhost:3000);
YouwillusetheusualMochadescribeanditcodeblocks.Youneedtofirst
set up some code that runs before any tests in a before block. This code
initializes Selenium and sets up which browser you want it to use. The after
blockdoesthecleanup.
Usingthedriverobject,youcangetaccesstoanyelementintheDOM,to
inspectitoraffectitinsomeway.Forexample,withanHTMLtextcontrol,you
cansenditkeystrokestoentervaluesasauserwould.Buttonscanbeclicked,
etc.
Manytimes,youdosomethinglikeabuttonclickandthenneedtowaitfor
the UI to respond. The driver has a wait() function you use to wait for UI
elementstobeavailable.Forexample,ifyouwerenavigatingtoanewpage,you
woulduseawaittoblockuntilthepagewasready.Thewait()functiontakesasa
firstparameter,whatyouarewaitingfor.Youcanthuswaitforanelementtobe
visible. Inside that function, you provide the id that will be used by the
webdriverobjecttolocatetheHTMLelement.Thereisatimelimitsetforhow
longitshouldwait.Itwillwaituptothattime,butiftheelementappearsbefore
that, it will immediately move on. Sometimes you will need to put in a
setTimeout()calltodothedelay.Thisshouldbeavoidedastheseaddupandadd
tothetotaltimeittakesforatestruntocomplete.
Here is the specifying of an id in your HTML that you then can use to
identifyabuttonbylaterinyourtestcode.
<buttonid="btnRegister"ng-click="register()">Register</button>
Aswith all Mocha tests, you need to call done()when that test is finished
andmoveontothenexttest.
ThewholepointofputtingtogetheraUIautomationtestwithSeleniumisto
mimic what you would normally do manually and thus have a repeatable test
suitethatyoucanrunaspartofaCI/CDscriptandsaveyoualotoftime.
ThereistheconceptofaUserAcceptanceTestorUATthatbasicallyisthe
userscriptthatyouwanttofollowtoprovethattheUIcandoeverythingitis
supposed to do. Development teams can create a UAT for each code iteration
theygothroughbeforeadeploymentcanbeapproved.
Youcansetupthetesttogoagainstyourstaging,productionorlocalhosted
site.HereisasmallsampleofthecodefortheUIautomationtestingthatuses
Selenium.IhavesetituptouseInternetExplorer,butyoucanuseanybrowser
youlike,evenahiddenone.Youwillgetamessageifyoudonothavethedriver
installed for the browser you want to use. For example, with IE, the file is
IEDriverServer.exe. You will see a message on where to get it from. You just
unzipitandcopyittoalocationthatisinyourpath.
NewsWatcherSeleniumtests
Thiscodesimplylaunchesthesiteandthenclicksonthelogintabandthen
attemptstologinwithauseremailthatdoesnotexist.Thecodeteststhatthe
correcterror message appears. Note how there needs to be delays put into the
testcodetowaitforUIchangestohappen.
describe('NewsWatcherUIexercising',function(){
vardriver;
varstoryID;
vartoken;
//Runsbeforealltestsinthisblock
before(function(done){
driver=newwebdriver.Builder().withCapabilities(
webdriver.Capabilities.ie()).build();
driver.get('http://localhost:3000');
driver.wait(webdriver.until.elementLocated(
webdriver.By.id('loginLink')),10000).then(function(item){
done();
});
});
//Runsafteralltestsinthisblock
after(function(done){
driver.quit().then(done);
});
it('shoulddenyaloginwithanon-registeredemail',function(done){
driver.findElement(webdriver.By.id('loginLink')).click();
driver.findElement(webdriver.By.id('formControlsEmail2')).
sendKeys('T@b.com');
driver.findElement(webdriver.By.id('formControlsPassword2')).
sendKeys('abc123*');
driver.findElement(webdriver.By.id('btnLogin')).click();
driver.wait(webdriver.until.elementLocated(
webdriver.By.id('currentMsgId')),5000);
//WaitafewsecondsfortheUItoupdate
setTimeout(function(){
driver.findElement(webdriver.By.id('currentMsgId')).getText().then(
function(value)
{
assert.equal(value,
'(Signinfailed:Error:Userwasnotfound.)');
done();
});
},5000);
});
});
Note:If you look at the selenium test code in my GitHub project, you will
find most of it commented out. This is because the code was taken from an
original Angular application and I decided not to continue with the Selenium
testingandinsteadgowithEnzyme.Seleniumstillhasitsplaceandisusefulfor
end-to-end user acceptance testing. You might want to investigate the use of
Nightwatch,asanicewrapperontopofSelenium.
18.2UITestingwithEnzyme
Enzyme is an API that you use to test your React components at the code
level. With it you will do component level testing. This is analogous to Unit
testingofservicelayercode.Youcanisolateagivencomponentandverifythat
itworks as an individual piece of code and then have a greater assurance that
whenconsumed,itwillallworktogether.
Youwillwanttostartwithtestsforcomponentsatthelowestlevelsbecause
itissometimesbesttousecodetotestcodeasitisthemostefficientmeansof
trying all the combinations of code paths (including error paths). As part of
doing this, you will mock data that might be coming from things like HTTP
calls.Thepointistoisolateacomponentasmuchaspossible.
Youcanalsotestcomponentsthatareatahigherlevel,suchastheonesthat
consume other components. This means you could even test the complete
applicationfromthehighestAppcomponentandexerciseallpartsofit.
ItisuptoyoutodecidehowrigorousyouwanttheEnzymeteststobe.For
example, you can test components and verify that when they are instantiated
theyhavetheproperstate,propsandstyles.Youcangofurtherandcodeuptests
foryour Redux reducers. Youcanalso do full testingas if you are interacting
withacomponentwithmouseclicksetc.
Node: Mocha is often used as the test runner for the Enzyme tests. I have
chosenJestinstead(builtontopofJasmine),becausethatiswhatisinstalledby
default with the React application that was created. It also has some great
featuresthatMochadoesnothave,suchasbuilt-incodecoverageandparallel
runningoftests.Youcan’treallytellthedifferenceanywaybetweenamochatest
fileandaJestone.Theybothhave‘describe’and‘it’blocksofcode.Bothalso
havetheabilitytousethedone()functiontodoasynchronoustesting.
EnzymeistheAPIyoumakeuseofinsideofeachtestcaseincodethatJest
willrun.Enzymeiseasytouse.Withit,youinstantiateacomponentwitheither
theshallow()ormount()functions.ShallowdoesnotcreateavirtualDOM,but
is lightweight and just creates the component at that level with no hierarchy
underneathit.Thisworksinmostcasesforyourtestingpurposes.
IfyouaretestingsomethinglikeacomponentthatiswrappedbyRedux,you
needtousethemount()capabilityasitwillgetlifecycleeventsrunning.Itwill
have a full virtual DOM, and a redux state store will be used. I will give
examplesofboththeshallowandthemountusage.
NewsWatcherEnzymetestswithshallow
Think back to the home page UI that had news stories served up. In that
React code, you can find a componentDidMount() function that goes to the
backendtofetchthelistoftopnewsstories.Thedatareturnedfromthefetchcall
getsplacedintothelocalcomponentstateandthentherender()functionupdates
theUIwiththelatestdata.Hereissometestcodeforverifyingallofthat,using
somemockeddata.Iplacedalotofcommentsinthecodetohelpexplainit.
importReactfrom'react';
import{shallow}from'enzyme';
import{createStore}from'redux';
importreducerfrom'../reducers';
importHomeNewsViewfrom'./homenewsview';
//AhelperfunctiontoputtogetheranHTTPresponseforourmocking
constmockResponse=(status,statusText,response)=>{
returnnewwindow.Response(response,{
status:status,
statusText:statusText,
headers:{
'Content-type':'application/json'
}
});
};
describe('<HomeNewsView/>(mockeddata)',()=>{
it('newsstoriesaredisplayed',(done)=>{
//Thisisthepayloadthatourmockingwillreturn
constmockData=[{
contentSnippet:"ThelaunchofanewrocketbyElonMusk’sSpaceX.",
date:1514911829000,
hours:"33hoursago",
imageUrl:"https://static01.nyt.com/images/blah.jpg",
link:"https://www.nytimes.com/2018/01/01/science/blah.html",
source:"Science",
storyID:"5777",
title:"RocketLaunchesandTripstotheMoon",
}];
//Weusetheactualreduxstorewithourofficialreducers
//WereplacetheJavaScriptfetch()functionwithourown
//andalwaysreturnourmockdata
conststore=createStore(reducer)
global.fetch=()=>Promise.resolve(mockResponse(200,null,JSON.stringify(mockData)));
//HereistheusageofEnzymetoinstantiateourcomponent
constrc=shallow(<HomeNewsViewdispatch={store.dispatch}/>,{disableLifecycleMethods:true})
expect(rc.state().isLoading).toEqual(true);
//Wearedoingshallowinstantiation,soweneedtocallbyhand
//thecomponentDidMount()functionandwaitonthepromiseresolve
//tothendotestingagainstwhatisexpectedbecauseofthe
//newsstoryfetchandrenderwiththatdata
rc.instance().componentDidMount().then((value)=>{
//Allstatepropertieswillbeupdatedbynow,
//however,therendermaynothavehappenedyet.
//Theupdate()callismade,sothetestofthe<h1>element
//willnowhavetherefreshedvalue
rc.update();
//Verifysomeofthestatethatwasset
expect(rc.state().isLoading).toEqual(false);
//verifysomeactualelementsthatwererendered
expect(rc.find('h1').text()).toEqual('HomePageNews');
constlistNews=rc.state().news;
expect(listNews.length).toEqual(1);
expect(listNews[0].title).toEqual('RocketLaunchesandTripstotheMoon');
expect(rc.find('b').first().text()).toEqual('RocketLaunchesandTripstotheMoon');
//VerifytheReduxstatewassetandthereis
//asuccessfulmessageonfetchcompletion.
expect(store.getState().app.currentMsg).toEqual('HomePagenewsfetched');
done();
})
});
});
You can see the use of shallow() and then the further usage of the object
returnedwithfunctionslikefind()andstate().Theseallowyoutointerrogatethe
instanceforwhatisexpected.
The expect() function of Jest is used for the validations. The component
returnedfromtheshallowusagecanbeusedtodothingslikefindsub-elements
andinspectthem.Youcanseewherewefindanh1elementandverifythetext
ofit.Youcanalsoinspectthestatepropertiesofthecomponent.Thisisdoneto
verifywhatnewsstorieswerefetched.
AnothertestisdonethatinspectstheReduxstorewithagetState()call.You
canlookatanythinginthestoreasyouseeishappeningintheverificationof
whattheapp.currentMsgissetto.
Since we are doing shallow rendering, the componentDidMount() function
willnotbecalledaspartofthelifecycleoperations.Thatiswhyyouseeitbeing
called in test code. The call to componentDidMount() is asynchronous, so we
neededtomakeuseofthedone()functiontotellJestthatthistestiscomplete.
Note:YoumightbetemptedtouseasetTimeout()intestcodeanddelayfora
fewsecondsuntiltheUIhasrendered.ThisactuallywillnotworkinJest,andis
abadideaanyway.Ifyouhadhundredsofteststhateachhaddelays,youwould
increasethetime ittakesto runyour testsandcould notreallybeguaranteed
anyraceconditionswouldworkout.
TheactualcomponentDidMount()componentcodegetscalled,soitwilldo
thingslikemakethefetchcallandthentryandalsosetthestateinReduxwitha
dispatchcall.Sincewewanttocontroltheactualreturnedresponse,weneedto
mockallofthisupandoverridethefetchfunction.WealsosetupReduxhere,
becausethatisnormallydoneintheAppcomponent,andweareworkingata
levelbelowthat.
Note:AnalternativetocallingthecomponentDidMount()wouldbeforyou
tomimicthecallsthatarehappeningtotheReduxactions.Youcouldplacecalls
tostore.dispatch()andthendosomeexpect()calls.Youcouldalsosetthestate
orpropsdirectlyoncomponentsandthenforceanupdateandthendotheexpect
tests.
NewsWatcherEnzymetestswithmount
The code to test the login must be a bit different. This is because the
componentisneedingRedux.Inthiscase,weneedtousethemount()capability
of Enzyme. We have to use mount() because we want to use the connected
component.Ithasthefunctionalitytousethereduxdispatchtoupdatepropsand
wecanverifythatthestorestateiscorrectlyset.
Hereissomecodethatmountsthecomponentandthenteststhataloginwas
successful.
it('Usercanlogin',(done)=>{
constmockData={
displayName:"Buzz",
userId:"1234",
token:"zzz",
msg:"Authorized"
};
conststore=createStore(reducer)
global.fetch=()=>Promise.resolve(mockResponse(201,null,
JSON.stringify(mockData)));
//Needtomockthelocalstoragecallaswell
constlocalStorageMock={
setItem:()=>{},
removeItem:()=>{}
};
global.localStorage=localStorageMock
constwrapper=mount(<ConnectedLoginViewstore={store}/>)
letrc=wrapper.find(LoginView);
expect(rc.props().session).toEqual(null);
rc.instance().handleLogin({preventDefault(){}}).then((value)=>{
expect(store.getState().app.session.displayName).toEqual('Buzz');
expect(store.getState().app.currentMsg).toEqual('SignedinasBuzz');
done();
})
});
Weoverridefetchagainasbefore.Noticehowwealsoneededtoprovidea
stubbed out preventDefault() call to not do anything, as handleLogin() is
expectingthatfunctiontoexist.
Note: We could have caused a click to have happened on the login button
and the code would have proceeded to execute that. The problem is that we
wouldnotknowwhenthatfinished,sowecansimplycallthefunctionourselves
andwaitfortheresolveanddowhattestingwewantafterthat.
Youwill notice we don’t even fill in the email and password, as our fetch
mockingdoesnotpayattentiontoanythingpassedinandreturnswhatwewant,
underourcontrol.
UIelementinteraction
Hereissomecodethatdoessomemanipulationoftheactualelementslike
thetextcontroltoholdtheemailandthecheckboxcontrol.Youcanuseshallow
ormount andaccomplish thesame thing. Whatwe are reallychecking hereis
thebindingofeachcontrolbylookingatthestatepropertiesthatshouldbesetas
aresultoftheinteractions.
it('Usercanchangeremembermecheckboxandenteranemail',()=>{
constrc=shallow(<LoginViewdispatch={()=>{}}session={null}/>,{disableLifecycleMethods:true
})
expect(rc.state().remeberMe).toEqual(false);
rc.instance().handleCheckboxChange({target:{checked:true}});
expect(rc.state().remeberMe).toEqual(true);
rc.find('#formControlsEmail2').last().simulate('change',{target:{value:"abc@def.com"}})
expect(rc.state().email).toEqual("abc@def.com");
});
Inthecaseofacheckbox,thereisnotawaytoclickitorcauseachangetoit
to have it run the onChange() handler so we just call the
handleCheckboxChange()methodourselves and fakewhat wouldbe passed in
fortheeventparameter.
Fortheemailtextinputelement,wecansimulatethetextentry.Inthiscase,
wecangetanonChange()tohappenautomaticallyforus.
GettingsetuptouseEnzymeandrunningtests
BeforeusingEnzyme,youneedtoinstallthenpmpackagesasfollows:
npminstall--saveenzymeenzyme-adapter-react-16react-test-renderer
ThenyoualsoneedtocreateafileinthesrcdirectorynamedsetupTests.js
withthefollowingcontents:
//src/setupTests.js
import{configure}from'enzyme';
importAdapterfrom'enzyme-adapter-react-16';
configure({adapter:newAdapter()});
Torunthetestsyoujustrunthefollowing:
npmruntest-react
Youcanalterthetestscriptinthepackage.jsonfiletoaddtherunningofthe
Enzymetestsandalsoaddaflagtotellittoincludecodecoveragenumbers.
"test": "mocha --timeout 30000 test/functional_api_crud.js && react-scripts test --coverage --
env=jsdom",
Ifyoudosomesearchingonlineandyouwillfindallkindsofcreativeways
to test with Jest and Enzyme. For example, I did not mention the ability of
EnzymetocompareDOMsnapshotsfromonetestruntoanother.
Goodtestscodecansometimesbeasdifficultorevenmoredifficultthanthe
actualproductcode.Ihaveworkedonseveralprojectswheretherewastwiceas
muchtestcodewrittenasproductioncode.
Codecoveragenumberslookasfollows:
Figure104-F12debuggerconsole
18.3DebuggingUICodeIssues
Tolocallydebugyouserversidenode.jscodeisverysimple.Byusingthe
VS Code editor, you have the capability to run it locally and set breakpoints.
Reactcode,however,runsinthebrowser.Browsershavetheirowndebugging
capabilities that you must learn and make use of. For example, with Chrome,
thereisaselectioninthemenuunderMoretoolstolaunchDevelopertools.
Onceopen,youcanclickontheConsoletabandseeerrorsthathappenin
yourcode.Thisimageshowsacodeerrorforanundefinedproperty:
Figure105-debuggerconsoleinChrome
This window is very handy to have open, to see problems you might not
otherwisenotice.Manytimes, itisquite easytogo andfixthe bugsyou find.
Other times, you will need to step through the lines of code to tell what is
happening.SimplyclicktheSourcestabandyoucanbrowseyourcodeandset
breakpoints in the browser. The experience is typical of any debugger, in that
youcaninspectvariablesandsetwatchesonthem.
ThisnextimageshowshowIusedthedebuggertosetabreakpointandthen
inspectedtheresponsedatabeingreturnedfromanHTTPrequest.Theleftpane
hasalistoffilesthatyoucanlookthrough.IopeneduptheJavaScriptcodefor
thehomenewsviewing.Inthepanewherethecodeis,youcanclickonaline
numberandsetabreakpoint.Youmayneedtohitthebrowserrefreshbutton,or
clickaroundinyourapplicationtomakethecoderun,tohityourbreakpoint.
Figure106-NewsWatcherHTTPrequestdebugging
YoucanhaveVSCoderunningthenodeserverandplacebreakpointsinthe
serversidecodeanddebugbackandforthinbothenvironments.
If you have written HTML, you know the capabilities of the browser
developertoolsandwillcertainlyknowhowtoinspecttheDOMelementsand
theirassociatedCSS.ClicktheDOMExplorertabtotrythisout.Forexample,
youmightfindthatanelementisnotstyledorfunctioningasexpected.Aquick
inspectioncanusuallyturnuptheissue.
TheNetworktabshowsyoutheHTTPtrafficgoingbackandforthsimilarto
what you see using Fiddler or Postman. The Performance tab can let you
capturethecallsgoingonandthenviewthatinacallgraphthatshowsthetime
ofeverythingindetails.Thisallowsyoutocheckforandfixperformanceissues.
TheMemorytabletsyousetuparunthatcapturesmemorysnapshotssoyou
caninvestigatememoryleaksinyourcode.
Figure107-ChromeNetworkcapturing
Chapter19:Server-SideRendering
The NewsWatcher application has been developed as a purely client-side
SPAapplicationthatloadsandrendersonaclientdevice.Itisruninthecontext
ofabrowser,whetherthatisonamobilesmartphone,tablet,laptoporpersonal
computer.Thisworksquitewell.
A SPA like NewsWatcher has great performance and mostly minimizes
traffic back and forth to just the data fetching. All the UI is always rendered
client-sideandisneverloadedfromtheserver,onceitisallbroughtover.There
isabitofresistanceinthedevelopmentcommunitytofullysanctionSPAsasthe
waytogo.Theirobjectionsarebasedontwopointsasfollows:
1. Performance is better servicing up pages from a server. A server
renderedappcancollectallthedataontheserversideandserveitup
fasterthanaSPA.Especiallywhenitcomestotheinitialpageloading,
ifthereisalotthatgoeswithit.
2. SEO (Search Engine Optimization) is more efficient for server
renderedHTMLpagesbecauseofhowGoogleindexesthem.
What some people are advocating, is to move to an SSR (Server Side
Rendering)designtoaddressthesetwoissues.Therecan,ofcourse,beahybrid
solutiontogiveyouthebestofaSPAandanSSRsolution.Let’sfirsttakealook
attheobjectionsonebyone.
Asfarasthefirstpoint,thiscanbemitigatedbyimplementingcodesplitting
so that not everything is sent to the client at the start. This is where you only
sendupUIasneeded.Thecodeiseffectivelysplitupintoparts.TheinitialUI
codeissentandrendersveryfastandthenotherpartsarebroughtinasneeded.
ThiswouldthenallowjustasfastofapageloadforaSPAasforanSSRsite.As
younavigatethroughtheUI,pagesarebroughttotheclientjustintime.
ThesecondpointisalsodebatableinthatGooglewillcrawlSPAapplications
andcanevenlookthroughJavaScriptcodewhiledoingwebindexing.
Note:ToreallydojusticetoanSSR,youalsoneedtoputinmetatagsinthe
HTMLashelpersforthewebindexerstohelpwithsearchengineoptimization
(SEO).
Ifyoudecideto,youcanwritethesitetobecompletelyrenderedasanSSR
site. You can even have Redux and React Router used on the server. As a
compromise,youcanmakeakindofhybridimplementation,wheresomeofthe
UIisrenderedontheserversideandreturnedasHTML/JavaScripttotheclient
browser and once there, that code can also contain React code and have that
code run on both the Browser side as well as on the SPA. This is what a
Universal application design pattern is all about being able to run the same
typeofcode(ReactRouter,Redux,etc.)ontheserverandontheclient.
ThehybridUniversalJavaScriptsolutionwouldcertainlyaddressbothissues
andmightbewhatyouwanttogotowards.Thiswouldmeanthatyouhavethe
initial page rendered with SSR, including its Redux store data along with the
initialpage so that it renders instantly. This gives you the speed for the initial
pagerendering.TheSEOissuewouldbe takencareofforthisinitialpage for
Googletoindex.Thepageloadsimmediatelyanddoesnothavetogobackto
fetchanyadditionaldatauntilfurtheruserinteractionsarehappening.
AUniversalhybridapproachisgreatifyouhaveaninitialroutepagewith
criticalinformationtobeindexed.Fromthere,thatpagecanevenfunctionasa
SPA,withsub-navigationthatgoesalongwithit.Youcandecidewhatpagesyou
want served up as SSR routes and then have each of those be SPA pages
functioningontheirown.
Note:ifyouarethinkingthatalotofyouruserswillbeonmobiledevices
that have cellphone or Wi-Fi connection anyway, you might as well develop
native applications. This means you have a purely native application they
download. This is in essence functioning like the SPA does, except that the
application is always instantly there on the device. You also get the benefit of
writingthenativeapplicationtoworkinamodewhereitisoff-line.Youcanalso
investigatewhataProgressiveWebapplicationisfordoingthatinthebrowser.
Youwillalsowanttoimplementsomecachingontheserversideifyouare
doing SSR. This is a huge performance advantage. For example, if the home
newspageisgoingtostaythesameforeveryoneforafewhours,yourenderit
andservethatupfromacachedcopyandthenupdateitwhenanewsetofnews
isreadytogo.ThereareNPMmodulesyoucanusethatmakethisseamlessand
areeasytoputintoyourcode.
19.1NewsWatcherandSSR
Idecidedtoexperimentwithhavingtheinitialhomenewspageasaserver-
side rendered page and then the rest of the application being a client SPA. It
turnedouttoworkreallywellandisanoptionforthosethatdecidetogothis
route. It may even turn out that someday create-react-app generates code that
allowsyoutheoptionofdoingcodeasSSR.
Note:YoucanfindmySSRexperimentonGitHub.Thiswasabriefattempt
andshouldnotbetakenasafinishedprojecttopatternanythingafter.Iwasable
to preserve the original app without doing an eject
(https://github.com/eljamaki01/NewsWatcher2RWebSSR). Be aware that I took
thecodesnapshotforthisexperimentatanearlystageofNewsWatcher,soalot
changedbetweenthatandtheotherrepositorythatendedupbeingthemainone
forthisbook.
What I did, is set things up for the route to the home news page to be
rendered with SSR. The rest of the application still works the same as a SPA.
TheideaisthataninitialcallcomesinfortheNewsWatchersiteandanHTML
page is rendered on the server side for it. This initial page is the home news
page. That HTML can also contain a special section that contains all of the
actualfetched data for the newsstories. This also goes up. Youmightwonder
whyintheworldthatwouldbegoingon.TheanswerhastodowithhowReact
works.
ReactwillonlyrenderUIthathaschanged.Thus,aninitialpageofthehome
newsisrenderedandthenReactalsodoestheworktoworkasifitisaSPAand
takesthedatasentupandalsotriestocreateavirtualDOMforthehomenews
fromthat.Itturnsouttobeidenticalwithwhatwasalreadybeingshowninthe
actualDOMandsoitthrowsitaway.
It turns out that the page instantly loads and there is no delay. It all just
instantlyappears.Otherviews,suchastheusernewspagewillshowtheloading
text while the componentDidMount() code chunks away, but the home news
pagenevershowstheloadingtext.
Onechangethatneedstohappenisthattheserver.jsfileneedstoserveup
theroute forthe main‘/’ routethat isthe UIapplication itself.That codewill
end up being in a new file that you create named ssrrender.js. Here is the
server.jsfilecodetoreturntheserver-renderedmainpage:
process.env.BABEL_ENV='production';
process.env.NODE_ENV='production';
require('babel-register')({
ignore:/\/(build|node_modules)\//,
presets:['env','react-app']
})
constSSRRender=require('./ssrrender');
constSSRRenderRouter=require('./ssrrenderRouter');
app.use('/',SSRRenderRouter)
//ServingupofstaticcontentsuchasHTML,images,CSSfiles,andJavaScriptfiles
app.use(express.static(path.join(__dirname,'build')));
app.use('/',SSRRender)
Thecodetorenderthestaticpageisgoingtolookfamiliar.Youwillevensee
the use of the Provider component and the Redux store and reducers. It is all
there as before. This means that the exact process of rendering a page on the
clientis now happening on theserver.Even the bootstrap style filesare there.
There are two interesting lines that do string replacements on the HTML
template. The first one does the actual replacement of the HTML that was
renderedbyReact.ThesecondoneistheonethataddstotheReduxstate.Itis
transferred with the news story list up to the client to be used there. This is
becausetheComponenbtDidMount()isnotusedtoretrieveitanymore.Hereis
allofthecode:
//ssrrender.js
constpath=require('path')
constfs=require('fs')
constconfig=require('./config');
constReact=require('react')
const{renderToString}=require('react-dom/server')
const{StaticRouter}=require('react-router-dom')
const{createStore}=require('redux')
const{Provider}=require('react-redux')
const{default:App}=require('./src/App')
const{default:reducer}=require('./src/reducers')
import{toHours}from'./src/utils/utils';
varexpress=require('express');
import'bootstrap/dist/css/bootstrap.css';
import'bootstrap/dist/css/bootstrap-theme.css';
varrouter=express.Router();
//ReturnalltheHomePagenewsstories.Verifywehavealoggedinuser.
module.exports=functionhandleSSR(req,res,next){
req.db.collection.findOne({_id:config.GLOBAL_STORIES_ID},
{homeNewsStories:1},
function(err,doc)
{
if(err)returnnext(err);
//Populateaninitialstate
for(vari=0;i<doc.homeNewsStories.length;i++){
doc.homeNewsStories[i].hours=toHours(doc.homeNewsStories[i].date);
}
letpreloadedState={homenews:{isLoading:false,
news:doc.homeNewsStories}}
//CreateanewReduxstoreinstance
conststore=createStore(reducer,preloadedState)
constcontext={}
consthtml=renderToString(
<Providerstore={store}>
<StaticRouter
location={req.url}
context={context}
>
<App/>
</StaticRouter>
</Provider>
)
//GrabtheinitialstatefromourReduxstore
constfinalState=store.getState()
//Sendtherenderedpagebacktotheclient
constfilePath=path.resolve(__dirname,'build','index.html')
fs.readFile(filePath,'utf8',(err,htmlData)=>{
if(err){
console.error('readerr',err)
returnres.status(404).end()
}
//We'regood,sosendtheresponse
letreplace1=htmlData.replace('{{SSR}}',html)
letreplace2=replace1.replace(`console.log("REPLACE")`,
`window.__PRELOADED_STATE__=${JSON.stringify(finalState).replace(/</g,'\\u003c')}`)
res.send(replace2)
})
});
};
Thereisoneminortweaktothehomenewsview.jsfile.Yousimplycomment
outthecomponentDidMount()function,becauseitalreadyhasthenewsstories
intheinitialloadandthatisplacedinthereduxstoretobeused.
The file index.js also has a minor tweak where it uses a special window
objectpropertywindow.__PRELOADED_STATE__propertythathasthenews
storiesandloadsthatintotheReduxstoreontheclientside.Youseethatonthe
server-sidecodeinssrrender.jsthatthestoreisalsopreloadedwiththatdata.You
canlookthereandyouwillfindtheactualcalltothedatabase.ThereisnoHTTP
call,asontheserversidewehavethedatabaseconnectiontousedirectly.
Note:Server-siderenderingisnotforeveryone.Whateveryoudo,youwould
havetoweighthecostsversusthebenefittohelpyoumaketherightdecision.It
canbetrickytogetrightandcancertainlymakeyourcodemorecomplicatedto
understandgoingforward.
Chapter20:NativeMobileApplication
developmentwithReactNative
The existing NewsWatcher React SPA client code can be taken and
transformedintocodethatrunsasanativeapplicationonamobiledevice.The
ReactWebappwasbuilttoberesponsive,meaningadapttothescreensizethat
it is running on. It was built using Bootstrap to give it such things as a
collapsiblemenuwhenrunningonamobilephoneinthebrowser.Thisstilldoes
notcomparetowhatperformanceandUIcapabilitiescanbeaccomplishedwith
atrulynativeapplication.
OnebenefitofNativeapplicationsisthattheycanbedesignedtoruninan
off-linemodewhennodataorWIFIconnectionisavailable.Sure,thereisthe
possibilityof doing a progressive web application, but why notjust stick with
thereal thing and go completely native? Native applications have much better
performancethanabrowser-basedapplicationrunningonaphone.Theotherbig
differentiatoristhatNativeapplicationscanaccessallthehardwarefeaturesof
the phone itself, such as the camera, contacts, gyroscope and many other
features.
Yousimplydoabitoftweakingtothecodeintherenderfunctionsandthen
useatooltocreateanativeapplicationfilethatisappropriateforeitherGoogle
PlayorfortheAppleAppstore.
AReactNativeapplicationrunsthelow-levelmobileplatformcodethrough
a JavaScript “bridge” layer to interact with the phone for accessing all its
functionality and presenting a UI. There is a JavaScript engine interface that
existsonbothiOSandAndroid.Thisisalow-levellayerthattheninteractswith
theOSforthephonetodothesamethingsthatotherplatformlanguagesonthe
phonesdo.YoumayhaveheardofObjective-C,Java,XcodeorSwift.
React web sites write to the HTML DOM, as that is the layer for all
browsers, but for phones, it is the JavaScript engine layer as the point of
interaction.Thismeansyougetthemostperformantapplicationspossible,and
accesstoallthephonesOSandhardwarefeatures.
ThegreatbenefitwithReactNativeasthecoretechnologyisthatthecode
can be very similar to that of your React web application. This is because the
React Native library is directly related to the web version. You have all the
familiar concepts, like the usage of JSX, lifecycle events, Component class,
rendermethod,usageofmoduleslikeReduxandmanyotherReactcapabilities.
TheonebigdifferenceyouwillrunintoisthatyouhavenoHTMLelements
availableinReactNative.No<div>,<span>,<ul>etc.Thereisalsoadifferent
approach to CSS for styling. All of this is kind of a pain, because you would
havegonethroughalotofworktogetallofthatwrittenupinyourReactWeb
application. It does not turn out to be too much of a hassle though. Partly
because you really should re-think your UI design when you are writing an
applicationforasmallmobiledevicescreen.Thewebsitepagesinmanycases
donotmakesensetoberenderedthesameonamobiledevice.Evenifyouare
doing a responsive design with libraries like Bootstrap, that is still not good
enough.
For now, the goal should be to take advantage of what you can from your
ReactWebapplication.Thenyoucanstrivetowriteyourmobilecodeonceand
beabletoruntheapplicationonbothiOSandAndroid.Itisnotalwayspossible
towritethecodeandrunitonbothiOSandAndroid,butyoucancomecloseif
youworkatit.
Note:Thisbookdoesnotcoverthetopicofhowtointegratelow-levelnative
codeintoyourapplication.Thisisatechniquewhereyouactuallywritecodein
Objective-CforiOSorJavaforAndroid.NewsWatcherdidnotrequirethistobe
done, and you would need to understand when and how you would need this
advancedtechnique.
20.1ReactNativestarterapplication
TheReactNativeapplicationrequiresacertainamountofscaffoldingtobe
inplaceinordertobeabletobebuiltandrun.Asoneoption,youcouldcopya
projectfromsomeoneelsefromGitHub.Therearealsotoolstogenerateastarter
application.ThesecondchoiceiswhatIwentfor.
WhatIdidwastorunacommandtogetmyReactNativeapplicationcreated
and then I brought over code from the web application version. I was able to
provehoweasyitistotaketheReactwebsitecodeandwriteanativeapplication
fromthat.Certainly,mostofthecodelogicisgoingtobethesame.Itismostly
theUImarkupthatwouldhavetochange.IwillnowgothroughallthestepsI
tooktogettheReactNativestarterapplicationupandrunning.
Step One: Install a command line utility tool and a Desktop
DevelopmentTool
With Node installed, you could use NPM to install the create-react-native-
app command line utility. This is used to create a starting point application.
Anotheroption would be to use the Expo npm utility that you can also install
fromnpm.IwilluseExpo.Expoisfullysupportedandevenmentionedonthe
officialReactNativesite.
HereisthecommandtogetExpoinstalled:
npminstallexp--global
The exp utility will be used to generate your starter application and
eventually create the finished application file that will be uploaded to the app
storesforotherstoinstall,suchasinGooglePlay.
Youwillalsoneedtoinstalladesktopapplicationthatactsasadevelopment
environment for Expo. It is called Expo XDE (Expo Development
Environment).ToinstallXDEyoucangototheexpowebsiteanddownloadthe
versionyouneed.TherearedownloadsdependingontheOSofyourmachine.
StepTwo:Createthestarterapplication
NowyoucancreatetheReactNativeapplication.Todothat,youwillusethe
Expo utility you just installed. Open a command prompt and create a folder
where you want the application to be created in. Then run the following
command,replacingthetextwithwhatyouwantyourapplicationcalled:
expinitmy-new-project
Alternatively,youcanlaunchtheXDEdesktopapplicationandfromamenu
selection, create an application. This is actually what I did. I selected the
templatethatcreatesanapplicationwithsomenavigationUIalreadyinplace.
Note: When you run XDE you will need to provide a username and
password. This is because XDE requires an account to be used for the
generationofthenecessaryfileforeventualuploadingtoanappstore.
StepThree:Runtheapplicationonyourmobilephone
WiththeXDEapplicationopen,youcanselecttohavetheapplicationyou
build to be run on your phone. Before doing that, you need to first configure
your phone for tethering and actually have it connected via USB to your
computer.
To set up tethering and debugging on an Android phone you can do the
following:
GotoSettingsandunderGeneralyoucanscrolluntilyoufindtheselection
forAboutphone.OpenthatandthentaponSoftwareinfo.TapBuildnumber
(youmayhavetotapitmultipletimes)andthenyouwillseeamessagethatyou
haveenableddevelopermode.
NowgototheGeneralsettingstotheDeveloperoptionsselection.Select
USBdebuggingtoturniton.
Youareallsettomakechangestosomecodeandexperimentandseewhat
you can do with React Native. I found that with my Windows machine, there
weretimesthattheconnectiontomyphonedidnotseemtoworktoowell.For
me,the“localhost”selection,fromthesettingsworkedthebestastheprotocol.
Figure108-Settingtheconnectionprotocol
Now,youcangototheXDEUIandopenyourprojectandthenyouwillbe
abletoclickontheDevicebuttonandselectOpenonAndroid.Theapplication
willloadandbestartedonyourphoneandyoucaninteractwithit.
Ifyourecall,therewerecommandlinescriptstoputtogethertheapplication
bundlefordeployment.WiththeusageofExpoXDE,allofthathappensasyou
openaprojectandclicktohaveitrunonyourphone.
Note: You can start up a phone emulator on your PC that can run the
applicationinit.Thisallowsyoutodothisinplaceofusingyourphysicalphone
that is connected via USB. You can create virtual devices for many different
typesofphones,includingiPhones.Todothatyouneedtoinstallsomethinglike
Xcode,AndroidStudio,ortheGenymotionemulator.
Thefollowingscreenshowsanentryforthetetheredphonethatyouwould
select to have your application run on. If you had an emulator running, the
emulator would also show up. You can also set up configuration to have the
applicationauto-loadanycodechangesthatarebeingmadeintheeditor.Thisis
calledhotreloading.
Figure109-ExpoApplication
20.2Components
React Native has a set of components you can use for doing things like
displaying text, getting input and displaying images. Besides this, there are
plenty of UI component libraries that you can install from NPM that can give
youwhatyouarelookingfor.Forexample,ifyoulikeMaterial-UIfromGoogle,
foryourlookandfeel,therearelibrariesthatimplementthatforReactNative.
Youcannowlearnaboutafewofthesimplecomponentsavailableandsee
thattheyareusedlikeanyothercomponentthatyouhaveseen.Youusethemin
arenderfunctionandpassneededpropstothemintypicalReactcodeyouhave
seenbefore.
TheViewandtheTextcomponentsaresomeeasycomponentstostartyour
learningwith.ViewisbasicallywhatadivwasinHTML.Itisacontainerfor
othercomponentstohelpwiththeUIlayout.
TheText component is for displaying text. The following code is close to
whatyougetwhenyoucreateaReactNativeApplicationfromExpo.Youcan
findafilenamesApp.jsthatisthestartingpointforrenderingyourUI.
importReactfrom'react';
import{Text,View}from'react-native';
exportdefaultclassAppextendsReact.Component{
render(){
return(
<Viewstyle={{borderTopWidth:30}}>
<Text>OpenupApp.jstostartworkingonyourapp!</Text>
</View>
);
}
}
Thiscodeshouldmostlylookfamiliar,asyouhavetheimportstatementsand
get the React usage from that, as well as the View and Text components. The
App uses the Component class and implements a render function that you see
returningtheViewwiththeTextinside.
Figure110–Simplestarterapplication
There are around three dozen components to use to construct your
application.Thesecan befound inthe onlinedocumentationfor ReactNative.
Thissectionwillonlycoverjustafewofthem.Youwillseemoreoftheusage
oftheseintheNewsWatchercode.
ThereisaTextInput component thatis used when you wantto collect text
from a user. It has a prop called onChangeText that you provide a callback
functionto,forcapturingthetextthatisentered.
ThereisaButtoncomponentanditisexactlywhatyouwouldthinkitis.It
hasanonPresspropthatyouprovideacallbackfunctionfor.Thiswillrunonthe
buttonpress.
There are other components for touch interaction, such as
TouchableHighlight, TouchableOpacity and TouchableElement. For example,
youmighthaveanimagethatcanbetouched,orabunchofitemsinalistthat
each have their own touch response. The onPress prop is used to provide a
callbackthatcanbeusedtoprocessthetouchevent.
Thereareseveralcomponentsyoucanusetodisplaylistsofinformation.If
you have a small set of data to render in a list, you can use the ScrollView
component. You will have a performance problem if the list has a lot of
elements.
There is a component named FlatList that only renders the data that is in
view as the user scrolls. You can also use the pagingEnabled prop for paging
throughdataandfetchingitondemandinbatchesasscrollinghappens.
There are many more components to investigate, such as for displaying
images,radiobuttons,checkboxes,selectionpickingandmuchmore.Readthe
official documentation to learn about all that is available
(https://facebook.github.io/react-native/docs/components-and-apis.html).
Note:Itmaysoonbepossibletohaveacommoncomponentlibrarythatcan
evenbeusedforyourWebSiteReactcodeandyourReactNativeapplication.
YouwillmostlikelystillneedtomakesomemodificationstotheUIdesignonthe
mobileapplicationbecauseofthescreensize.
20.3Stylingyourapplication
As was mentioned, HTML elements are not available to use in a React
Native application. CSS is also not available. Instead of using CSS, there is a
specialwayofapplyingstylerightintheJavaScriptyouwrite.Todothestyling,
each core components of React Native all accept a prop named style. This is
closeto how CSS works on theweb, however, names are writtenusing camel
casing.Forexample,youusebackgroundColorinsteadofbackground-color.
While you can use a JavaScript object right inside of the prop, it is much
cleanertosplitthisoutandusetheStyleSheet.createfunctiontodefineseveral
stylesinoneplace.Youthenreferencethatinyourrenderfunctioncode.Hereis
an example that shows the embedded object and the usage of the function
mentioned:
importReactfrom'react';
import{StyleSheet,Text,View}from'react-native';
exportdefaultclassAppextendsReact.Component{
render(){
return(
<View>
<Viewstyle={{width:50,height:50,backgroundColor:'blue'}}/>
<Textstyle={styles.myTextStyle}>Thisissometexttoexperimentwith</Text>
</View>
);
}
}
conststyles=StyleSheet.create({
myTextStyle:{
color:'blue',
fontWeight:'bold',
fontSize:12,
},
});
Widthandheight aresetwith propertiesand usevaluesthat willcausethe
sizingtobe thesameacross devices,regardless ofresolution.There isa View
componentusedto drawablue rectangle.Therectangle areais50 by50. The
Text component brings in the style from the returned object of the
StyleSheet.create()call.Therectangleandthetextarebothblue.Thefollowing
imageshowshowthiswilllook.
Figure111-Styling
20.4Layoutwithflexbox
Flexbox is a standard that exists for web page layout. You can read the
Mozilla(MDN) documentation to learn all the fine details. React Native takes
thisstandardandmakesuseofitwithinitsstylingcapability.Whatthisadds,are
propertiesyoucanaddtoyourstylingobjectthatspecifyexactlayoutareas.
Thefirstpropertytoknowaboutistheflexproperty.Thisiswhatyoucan
use to specify relative sizing of anything. For example, you can have three
rectanglesandgivethemeachaflexvalueof1andthenrelativetoeachother,
theywillgetsized.Sincetheyeachhaveavalueofone,theyeachtakeupthe
sameamountofspace.
Thislayout withflexbox can behierarchical, meaning thata parent canbe
part of a flex layout at its level, and then inside of that there, is another flex
layoutthatisapplied.
By default, the flex layout is determining the vertical layout of the area
(called column layout). You can alter that and make it specify the horizontal
layout(calledrowlayout).Youcanalsospecifythejustification,alignmentand
spacing.
Hereissomesamplecodethathastwosubsections.Onehasaflexvalueof
twoandtheotheravalueofone.Thismeansthatthetopareawillbetwicethe
sizeofthelowerarea.Studythefollowingcode:
importReactfrom'react';
import{StyleSheet,Text,View}from'react-native';
exportdefaultclassAppextendsReact.Component{
render(){
return(
<Viewstyle={styles.container}>
<Viewstyle={styles.subContainer1}>
<Textstyle={styles.childText}>One</Text>
<Textstyle={styles.childText}>Two</Text>
<Textstyle={styles.childText}>Three</Text>
</View>
<Viewstyle={styles.subContainer2}>
<Textstyle={styles.childText}>Four</Text>
<Textstyle={styles.childText}>Five</Text>
<Textstyle={styles.childText}>Six</Text>
</View>
</View>
);
}
}
conststyles=StyleSheet.create({
container:{
flex:1,
marginTop:25
},
subContainer1:{
flex:2,
flexDirection:'row'
},
subContainer2:{
flex:1
},
childText:{
flex:1,
borderWidth:2,
textAlign:'center',
fontSize:16
},
});
YoualsoseethatoneoftheflexboxareashasaflexDirectionsetto‘row’.
Thismakesthecomponentsinsideofitbespreadouthorizontally.Hereiswhat
theresultlookslike:
Figure112-Flexboxlayouthierarchy
Therearemanymoreflexboxstylingpropertiesyoucantakeadvantageof.
Themainpointofflexboxistogetyouawayfromspecifyingpixeldimensions
for anything and having everything sized relative to each other. That way no
matterthedevice,itcanbedisplayedproperly.
20.5Screennavigation
ReactNativedoesnothaveabuilt-incapabilitytosetupanykindofscreen
transitions. Instead, there is a very popular npm library that you can install to
handlethis.Youcandoannpminstallofreact-navigationandthenusethattoset
up different types of navigation. For example, if you want to treat your
navigationlikeastack,whereyoucangofromonescreentoanotherandasyou
transition,thepreviousscreensstayinthestackhistorythatyoucangobackto.
Here is a simple example that shows having a screen with a clock on it and
anotherscreenthatshowstheweatheronit.
import{StackNavigator}from'react-navigation';
constApp=StackNavigator({
Clock:{screen:ClockScreen},
Weather:{screen:WeatherScreen},
});
The ClockScreen and WeatherScreen are just components with a render
function to display whatever you can imagine for each of those screens. The
importantcodetoseeissimplythesettingupoftheuseoftheStackNavigator.
Thisallowsyoutohavecodethatcallsanavigatefunctiontogotoeitherofthe
screensandevenpassinparameters.ThebackbuttononanAndroiddevicewill
alsotakeyoubackascreeninthestack.
The header provided by StackNavigator will contain a back button to go
backfromtheactivescreenifthereisone.
The other mode of navigation is that of using tabs. You see this type of
navigationallthetimeinmobileapplications.Youwillseethetabsatthetopor
thebottom.Thecodeisexactlythesame,butinsteadofusingStackNavigator,
youuseTabNavigator.Once youdoso,you willseethetabappearfor youto
makeyourscreenselection.Youcanaddstylingtothistoaddicons.
The navigation can be set up to be nested. For example, you can have the
main navigation with tabs, but within each screen for the tab, you can set up
somestacknavigation,suchashavingatabwithalist,andclickingonanitem
inthelistdoingastacknavigationtoadetailsscreen.
20.6Devicecapabilityaccess
TheReactNativeAPIisstillgrowinginitscapabilities.Thisistrueasfaras
being able to access the capabilities of the phone hardware. For example, you
canaccessthephone’scamerarollofphotos,butnotthecameratotakeaphoto.
Youcangetatthegeolocation,butnotattheaccelerometer.Itcanbeassumed,
thatovertime,mostphonecapabilitieswillbeexposedthroughtheReactNative
API.
Until that time, you can simply use the API provided by Expo. Besides
providingtheUItocreateandrunapplications,therehasbeenprovidedalibrary
in npm that you can use in your code to access thing like the camera,
accelerometer,fingerprint,gyroscope,contactsandmuchmore.
TousetheCameracomponent,youuseitlikeotherReactcomponents.This
meansthatyouwillneedtorenderitasanelementfirstinsomeJSX.Toactually
take a photo, you need to get a reference to the component and then call a
functiononit.Hereiswhatitwouldlooklike.
//intherender()function
<Cameraref={ref=>{this.camera=ref;}}/>
//Somewhereincode,perhapsinresponsetoatouchevent
this.camera.takePictureAsync().then(data=>console.log(“Tookaphoto”));
20.7CodechangestoNewsWatcher
You can now learn about the code changes that are needed to get the
NewsWatcherapplicationwrittenasanativeapplication.
Thestartingpoint
AswithourotherReactcode,thiscodealsohasanApp.jsfilethatactsasthe
startingpointforyourlogictorenderandrunallyourcode.
Thecodewillsetupthenavigation,includingtheUIforthenavigationbar
andalltheindividualscreencomponentsthatwillmakeuptheapplication.The
navigationbar is dependanton the mobile platform,because iOS andAndroid
UI navigation can look different.Thus, there is code that determines which to
render.ThePlatform.ospropertyholdsastringthatwilltellyouwhichmobile
platformyouarerunningon.
YouwillseeReduxbeingusedintheeactsamewayasbefore.Hereisallof
thecodeforthestartup.
//App.js
importReactfrom'react';
import{Platform,StatusBar,StyleSheet,View,AsyncStorage}from'react-native';
import{AppLoading,Asset,Font}from'expo';
import{Ionicons}from'@expo/vector-icons';
import{TabNavigator}from'react-navigation';
importMainTabNavigatorfrom'./navigation/MainTabNavigator';
import{createStore}from'redux'
import{Provider}from'react-redux'
importreducerfrom'./reducers'
import{fetchMyNews}from'./utils/utils';
import{fetchMyProfile}from'./utils/utils';
conststore=createStore(reducer);
constRootNavigator=TabNavigator(
{
Main:{
screen:MainTabNavigator,
},
},
{
navigationOptions:()=>({
headerTitleStyle:{
fontWeight:'normal',
},
}),
}
);
exportdefaultclassAppextendsReact.Component{
state={isLoadingComplete:false};
componentDidMount(){
//Checkfortokenindevicelocalstorage,meaninguserissignedin
AsyncStorage.getItem('userToken',function(err,value){
if(value){
consttokenObject=JSON.parse(value);
store.dispatch({ type: 'RECEIVE_TOKEN_SUCCESS', msg: `Signed in as
${tokenObject.displayName}`,session:tokenObject});
fetchMyNews(store.dispatch,tokenObject.userId,tokenObject.token);
fetchMyProfile(store.dispatch,tokenObject.userId,tokenObject.token);
}
})
}
render(){
if(!this.state.isLoadingComplete&&!this.props.skipLoadingScreen){
return(
<AppLoading
startAsync={this._loadResourcesAsync}
onError={this._handleLoadingError}
onFinish={this._handleFinishLoading}
/>
);
}else{
return(
<Providerstore={store}>
<Viewstyle={styles.container}>
{Platform.OS==='ios'&&<StatusBarbarStyle="default"/>}
{Platform.OS==='android'&&<Viewstyle={styles.statusBarUnderlay}/>}
<RootNavigator/>
</View>
</Provider>
);
}
}
_loadResourcesAsync=async()=>{
returnPromise.all([
Asset.loadAsync([
require('./assets/images/robot-dev.png'),
require('./assets/images/robot-prod.png'),
]),
Font.loadAsync({
//Thisisthefontthatweareusingforourtabbar
...Ionicons.font,
'space-mono':require('./assets/fonts/SpaceMono-Regular.ttf'),
}),
]);
};
_handleLoadingError=error=>{
//Inthiscase,youmightwanttoreporttheerrortoyourerror
//reportingservice,forexampleSentry
console.warn(error);
};
_handleFinishLoading=()=>{
this.setState({isLoadingComplete:true});
};
}
conststyles=StyleSheet.create({
container:{
flex:1,
backgroundColor:'#fff',
},
statusBarUnderlay:{
height:24,
backgroundColor:'rgba(0,0,0,0.2)',
},
});
NavigationUI
The react-navigation module is used that requires an object as part of its
construction.Therearedifferenttypesofnavigationsavailable,suchasstackand
tab. We will be using tab by using the TabNavigator function. You pass in an
objecttothatandspecifythefilesthatholdthescreenUIsandalsothetabsand
iconsforthetoolbartouseforeach.
HereisthecodefortheconfiguringofthenavigationfortheNewsWatcher
application:
//MainTabNavigator.js
importReactfrom'react';
import{Platform}from'react-native';
import{Ionicons}from'@expo/vector-icons';
import{TabNavigator,TabBarBottom}from'react-navigation';
importColorsfrom'../constants/Colors';
importHomeScreenfrom'../screens/HomeScreen';
importMyNewsScreenfrom'../screens/MyNewsScreen';
importLoginScreenfrom'../screens/LoginScreen';
importProfileScreenfrom'../screens/ProfileScreen';
exportdefaultTabNavigator(
{
HomeNews:{
screen:HomeScreen,
},
MyNews:{
screen:MyNewsScreen,
},
NewsFilters:{
screen:ProfileScreen,
},
Account:{
screen:LoginScreen,
},
},
{
navigationOptions:({navigation})=>({
tabBarIcon:({focused})=>{
const{routeName}=navigation.state;
leticonName;
switch(routeName){
case'HomeNews':
iconName=
Platform.OS==='ios'?
`ios-home${focused?'':'-outline'}`
:'md-home';
break;
case'MyNews':
iconName=
Platform.OS==='ios'?
`ios-funnel${focused?'':'-outline'}`
:'md-funnel';
break;
case'NewsFilters':
iconName=
Platform.OS==='ios'?
`ios-options${focused?'':'-outline'}`
:'md-options';
break;
case'Account':
iconName=
Platform.OS==='ios'?
`ios-person${focused?'':'-outline'}`
:'md-person';
}
return(
<Ionicons
name={iconName}
size={28}
style={{marginBottom:-3}}
color={focused?Colors.tabIconSelected:Colors.tabIconDefault}
/>
);
},
}),
tabBarComponent:TabBarBottom,
tabBarPosition:'bottom',
animationEnabled:false,
swipeEnabled:false,
}
);
TheiconNamepropertycontainsavaluespecifyingthevectorgraphicicon
touse.TheseneedtobesetforvaluesdependingoniOSandAndroid.Thereisa
website to go to for seeing the list of possible icons. You can search the site
https://expo.github.io/vector-icons/.
NowyoucanconsidereachofthescreensthataresimplyReactComponent
derivedclassesthatReactNativenavigationusestorenderastheyarebrought
intousage.
TheHomeNewsscreen
Thisiscodeforshowingthehomepagescreenthatcontainsthedefaulttop
news stories. You will see much that is familiar. You see that there is still a
componentDidMount()function that contains the backend fetch call.The local
componentstateisusedforthatstorage.
In the scrollable list of news stories, you see something new. This is the
TouchableNativeFeedback React Native component. This is used as the outer
parentforallofwhatiscontainedinsideandgivestheabilitytocaptureatouch
thatwilllaunchthebrowsertodisplaythenewsstory.Everythingelseisafairly
straightforward translation from the HTML elements to the ones that are
supportedbyReactNative.
//HomeScreen.js
importReactfrom'react';
import{
Image,Platform,StyleSheet,
Text,TouchableHighlight,
TouchableNativeFeedback,View,
ScrollView,Alert
}from'react-native';
import{WebBrowser}from'expo';
import{toHours}from'../utils/utils';
exportdefaultclassHomeScreenextendsReact.Component{
constructor(props){
super(props);
this.state={isLoading:true,news:null};
}
staticnavigationOptions={
title:'HomeNews',
};
componentDidMount(){
returnfetch('https://www.newswatcher2rweb.com/api/homenews',{
method:'GET',
cache:'default'
})
.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))
.then(response=>{
if(!response.ok||response.status!==200){
thrownewError(response.json.message);
}
for(vari=0;i<response.json.length;i++){
response.json[i].hours=toHours(response.json[i].date);
}
this.setState({
isLoading:false,
news:response.json
});
})
.catch(error=>{
this.props.dispatch({type:'MSG_DISPLAY',
msg:`HomeNewsfetchfailed:${error.message}`});
Alert.alert(`HomeNewsfetchfailed:${error.message}`);
});
}
onStoryPress=(story)=>{
WebBrowser.openBrowserAsync(story.link);
};
onNYTPress=()=>{
WebBrowser.openBrowserAsync('https://developer.nytimes.com');
};
render(){
if(this.state.isLoading){
return(
<View>
<Text>Loadinghomepagenews...</Text>
</View>
);
}
letTouchableElement=TouchableHighlight;
if(Platform.OS==='android'){
TouchableElement=TouchableNativeFeedback;
}
return(
<View>
<ScrollView>
{this.state.news.map((newsStory,idx)=>
<TouchableElementkey={idx}
onPress={()=>this.onStoryPress(newsStory)}>
<Viewstyle={styles.row}>
<Viewstyle={styles.imageContainer}>
<Imagesource={{uri:newsStory.imageUrl}}
style={styles.storyImage}/>
</View>
<Viewstyle={styles.textContainer}>
<Textstyle={styles.storyTitle}numberOfLines={2}>
{newsStory.title}
</Text>
<Textstyle={styles.storySnippet}numberOfLines={3}>
{newsStory.contentSnippet}
</Text>
<Textstyle={styles.storySourceHours}>
{newsStory.source}-{newsStory.hours}
</Text>
</View>
</View>
</TouchableElement>
)}
<TouchableElementkey={this.state.news.length}
onPress={()=>this.onNYTPress()}>
<Viewstyle={styles.row}>
<Image
source={require('../assets/images/poweredby_nytimes_30b.png')}
/>
<Viewstyle={styles.textContainer}>
<Textstyle={styles.storyTitle}numberOfLines={2}>
DataprovidedbyTheNewYorkTimes
</Text>
</View>
</View>
</TouchableElement>
</ScrollView>
</View>
)
}
}
conststyles=StyleSheet.create({
row:{
alignItems:'center',
backgroundColor:'white',
flexDirection:'row',
borderStyle:'solid',
borderBottomColor:'#dddddd',
borderBottomWidth:StyleSheet.hairlineWidth,
padding:5,
},
imageContainer:{
backgroundColor:'#dddddd',
width:90,
height:90,
marginRight:10
},
textContainer:{
flex:1,
},
storyImage:{
width:90,
height:90,
},
storyTitle:{
flex:1,
fontSize:16,
fontWeight:'500',
},
storySnippet:{
fontSize:12,
marginTop:5,
marginBottom:5,
},
storySourceHours:{
fontSize:12,
color:'gray',
},
});
HereiswhatthatUIlookslikeonmyphonefothehomenewsscreen:
Figure113–Homenewsscreen
IfyouarelookingcloselyandcomparingthecodeoftheReactWebappwith
that of the React Native app, you will notice that we don’t actually use the
componentDidMount() function in the React Native code in some places. You
willseethattheProfileScreenandMyNewsScreencomponentsdon’thavethat
functionliketheHomeScreencomponentdoes.
Thereareafewreasonswhy.Onedifferenceisthatthisgetscalledonlyonce
and it gets called even before you switch to each screen. For example, it gets
calledontheprofilescreenatthestartuptimeoftheappeventhoughtheprofile
screenisnotbeingshown.Thefetchneedstomakeuseofthesessiontokenat
thattimetoget dataforthe user.The problemisthat thereisarace condition
with the App.js that gets the token from the local storage and it might not be
available yet. The fix is to place the fetching when you know the session is
available.
TheHomeScreencomponentdoesnotneedaJWTtokentocalltheendpoint
APIitisusing,sothereisnotimingissue.
Theotherscreens
Similarcodetransformationshappenformakingtheotherscreens.Youcan
look at the code for the LoginScreen, MyNewsScreen and ProfileScreen
components.Iwillgiveafewhighlightsandgiveyouscreenimagestolookat.
Modal:YouwillseetheusageoftheModalcomponent.Inthiscase,itisa
bitdifferentfromwhatwedidintheReactwebcode.Here,thereisactuallya
visiblepropthatwesettocontrolifitisvisibleornot.
Buttons:YouwillseeinthecodethatIexperimentedwithmakingbuttons
using both the Button and the TouchableOpacity components. The latter is
simplymoreversatile,butbothbehaveandlooksimilar.
HereiswhatthatUIlookslikeonmyphoneforamodalcomponentandthe
buttons:
Figure114–ModalUI
Keyboard:TheKeyboardAvoidingViewcomponentisusedtowrapUIthat
willmoveoutofthewaywhenakeyboardinteractionUIismovedintoplace.
This is used around the TextInput components. The TextInput themselves also
have some interesting props to use to do things like stating what type of
keyboardtodisplayandhowtotransitiontoanotherfield.
HereiswhatthatUIlookslikeonmyphoneforthekeyboardusage:
Figure115–KeyboardUI
Picker:ThePicker component is used to display a type of drop-down for
makingselections.Theselections,inthiscase,arethenewsfiltersthatyoucan
setupandchoosebetween.
HereiswhatthatUIlookslikefortheprofilescreenwheretheuserspecifies
theirnewsfilters:
Figure116–NewFilterprofileUI
Applicationconfigurationfile
ThereisaspecialfilethatisprovidedthatcontrolssomeaspectsoftheReact
Nativeapplication. Youcan openand look at the app.jsonfile to view whatit
contains.Youcanseethat youspecifythingslikethesplash screenimageand
alsowhatthedescriptionisfortheapplicationasshownintheappstore.Oneof
the things I needed to do for the Android section was to specify the
WAKE_LOCKpermission.Thisissettokeepthescreenfromdimmingandthe
processorfromgoingtosleep.
//app.json
{
"expo":{
"name":"NewsWatcher",
"description":"Newsservingapplication",
"slug":"NewsWatcher",
"privacy":"public",
"sdkVersion":"23.0.0",
"version":"1.0.0",
"orientation":"portrait",
"primaryColor":"#cccccc",
"icon":"./assets/images/icon.png",
"splash":{
"image":"./assets/images/splash.png",
"resizeMode":"contain",
"backgroundColor":"#ffffff"
},
"ios":{
"supportsTablet":true
},
"android":{
"versionCode":2,
"package":"com.blueskyproductions.newswatcher2rweb",
"permissions":["WAKE_LOCK"]
}
}
}
20.8ApplicationStoreDeployment
Aftertheapplicationisalldeveloped,youwillwanttomakeitavailablein
oneoftheapplicationstores.Iwillwalkthroughsomeofthedetailsyouneedto
knowtogetthisupandontheGooglePlaystore.Whatyouwilldoiscreatethe
APKstandalonebinarythatisusedtodistributeyourapplication.
When using Expo, the standalone app knows to look for updates at your
app's published URL. You can later run a command (exp publish) to upload a
new version. The next time a user opens the app they will automatically
downloadthenewversion.Thisiscalledan"OvertheAir"(OTA)updateandis
builtintoExposoyoudon'tneedtoinstallanything.
To build the APK (or iOS IPA file), open a Node.js command prompt. A
regular windows DOS prompt is not capable of running the commands. You
needtorunthefollowingcommandinthedirectoryofyourproject.
expbuild:android
This process will transfer the application code up to a cloud server run by
Expoandabuildprocesswillrunthere.Youcanseethatitisprogressingifyou
go to the link provided for you each time. Once a build is done, you can also
install it directly from the Expo portal with the Expo app on your phone by
snappinganimageoftheQRcodeyoucanview.
Figure117-Buildoutput
It could take five or so minutes to complete. Once it is done, you are
presentedwithaURLtousetodownloadtheAPKfileinthecaseofanAndroid
build.YourunasimilarcommandtocreateafileforiOS.
Onceyouhavedownloadedthefile,youcanuploadittotheappstore.Toget
thatupandavailabletoAndroidusersthroughtheGooglePlaystoreyouneedto
haveanaccounttodosoandgototheportaltoworkthroughthat.Onceyour
application is available in the store, you can see it at
https://play.google.com/store and of course open thePlay Store on your phone
andinstallit.
To upload your application file for Android and publish it on Google Play
youcangotohttps://play.google.com/apps/publish.Youcanclickthebuttonto
create a new application. You can begin by publishing your app as an Alpha
release.YoucanclickonReleasemanagementonthe leftmenuare andthen
chooseAppreleases.Thenonthatscreen,youwillseewhereyoucanclickto
managetheAlpharelease.Youcancreateareleaseanduploadyour.APKfile.
You can continue to upload newer updates to your APK file as you add
features and fix bugs. Alternatively, Expo can be used to publish releases that
don’trequireyoutogothroughtheGooglePlayportal,butareupdatesthatthe
applicationwillautomaticallydownloadonthemobiledevice.
OnceyouhavetheAPKfileuploadedtoGooglePlay,youcanclickonthe
Alpharelease and add testers to it. Youcan email a specificURLthat will be
availabletoanyfriendsthatwouldbeabletohelptesttheapplication.Youcan
then promote it to Beta and finally to Production where it will be seen by
everyoneintheGooglePlayapplicationstore.
If you have an Android phone, you can open Google Play and search for
“NewsWatcher”andinstalltheapplicationIuploadedtherealready.
Note:TherearestillafewkinkstoworkouttogetEnzyme,withJesttobe
abletodoamountandshallowrenderingintestcodewiththem.Thatwouldbe
agreatresearchprojectonitsown.ThereisdocumentationinEnzymeforthis,
butitstilldoesnotcarryitdeepenoughtobeabletodotestingofReactNative
asitdoeswithReactfortheweb.
LettingGooglePlaymanagethepublicsigningkey
ThereisonethingIomittedtomakeclearintheprevioustext.Thatisthatif
you upload the application with the Expo signed key, you can't ever upload
anotherAPKfile,asGooglePlaywillcomplainaboutanumberofthings.For
example,it will complain that the same versionnumber is being used already.
Butifyouchangethat,thenExpogenerates anewkeyforitand GooglePlay
willnotacceptit.ReadtheExpodocumentationonthisandtheymakeitclear
thatyoucanoptintoGooglePlaydoingtheusersignedkeyandthenyoucanset
upExpotoletyoumanagesigningwithaprovideduploadkeythatyougiveit.
Bottomline,itseemslikeyouwouldwanttogowithGooglePlaymanaging
thesigningsothatyoucanuploadnewAPKfiles.Otherwise,youaretiedinto
using Expo always to do the updates in the app through their mechanisms. I
knowthisallsoundsconfusingatfirst,sojusttryandfollowalonghere.
To start with, you need to open Android Studio and just create a new
application. Then once that is created, click on the Build menu and select
GenerateSignedAPK.TherewillbeabuttontherenamedCreateNew.Click
thatandfillintheform,suchasfollows:
Figure118–Creatinganuploadsigningkey
Follow through and do the APK generation. You will not be needing the
APKfile, butwill be takingthe jks fileand using thatwhen you dotheExpo
build.Makesuretotakenoteofthepasswordsandthealiasyoutypedin.Now,
whenyouruntheExpobuildcommandandselecttoprovideyourownkey,you
willbeaskedforthepathandnameofthejksfile.Followthepromptsandyou
willgetanAPKfileyoucanusewithGooglePlaynow.Youwillusethisfile
overandovertodothesigningeverytimeyouwanttouploadanewversionof
yourapplicationtoGooglePlay.
Note:IutilizeExpointhisbookbecauseitistheeasiestandfastestwayto
getsomethingupandworking.Youmaydecidethatyouwantthefullflexibility
tocreateandmanageyourReactNativeapplicationyourself.Inthatcase,you
shouldcreate the applicationwith create-react-native-app installedwith NPM.
Youinstallthisandcandoeverythingyouneedfromthecommandline,oreven
withtheuseofAndroidStudio.
CONCLUSION
A lot of material has now been covered and you should at this time feel
really good about your newly acquired knowledge and skills. You started by
learningaboutwhatathree-tierarchitectureisandhowaMERNJavaScriptfull-
stackimplementationisagreatchoiceforanimplementation.
TheNewsWatchersampleapplicationwasfeaturedthroughoutthebookasa
fullend-to-endimplementationofthearchitectureyouhavebeenlearningabout.
Younowhavethecodethatyoucanrefertoandbaseanyofyourownworkon.
Thethree partsof the book coveredthe layers ofa three-layerarchitecture
andgaveyoufoundationalknowledgeabouteachlayer.Youalsolearnedabout
theSDKstouseandspecificusagescenariosofeverythingfortheNewsWatcher
application.
You can now go forward and do your own learning for your own specific
needsandbesuccessful.Youmaychoosetojustworkinone ofthelayers,or
you may be able to contribute across all layers. It is always good to be
knowledgeableoverallasyouwillknowbetterhoweverythingworksinternally
andrealizehowwhatyoudoinonelayeraffectsanotherlayer.